[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 14:04:20.070160 11301 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.9.126:41133,127.11.9.125:33259,127.11.9.124:39325
I20240502 14:04:20.071635 11301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:04:20.072607 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:20.088717 11307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:20.089985 11308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:20.090193 11301 server_base.cc:1034] running on GCE node
W20240502 14:04:20.089175 11310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:21.189775 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:04:21.190001 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:04:21.190184 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658661190160 us; error 0 us; skew 500 ppm
I20240502 14:04:21.190963 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:04:21.200250 11301 webserver.cc:460] Webserver started at http://127.11.9.126:38245/ using document root <none> and password file <none>
I20240502 14:04:21.201079 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:21.201290 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:21.201761 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:21.207921 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-0-root/instance:
uuid: "115381676a2a4976a81a8a78fd29a261"
format_stamp: "Formatted at 2024-05-02 14:04:21 on dist-test-slave-j72s"
I20240502 14:04:21.215468 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20240502 14:04:21.220414 11316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:21.221452 11301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:04:21.221807 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-0-root
uuid: "115381676a2a4976a81a8a78fd29a261"
format_stamp: "Formatted at 2024-05-02 14:04:21 on dist-test-slave-j72s"
I20240502 14:04:21.222146 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:21.293972 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:21.295454 11301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:04:21.295931 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:21.362234 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.126:41133
I20240502 14:04:21.362318 11367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.126:41133 every 8 connection(s)
I20240502 14:04:21.368045 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 14:04:21.368945 11368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 14:04:21.374963 11370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:21.377606 11371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:21.379858 11301 server_base.cc:1034] running on GCE node
W20240502 14:04:21.380779 11373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:21.381498 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:04:21.381717 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:04:21.381906 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658661381884 us; error 0 us; skew 500 ppm
I20240502 14:04:21.382436 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:04:21.386356 11301 webserver.cc:460] Webserver started at http://127.11.9.125:40161/ using document root <none> and password file <none>
I20240502 14:04:21.386818 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:21.386992 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:21.387241 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:21.388228 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-1-root/instance:
uuid: "54cf04c1f8cb4ae8918bd36f3b9a21c5"
format_stamp: "Formatted at 2024-05-02 14:04:21 on dist-test-slave-j72s"
I20240502 14:04:21.390961 11368 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.9.126" port: 41133 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:04:21.393507 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20240502 14:04:21.399040 11380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:21.399814 11301 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20240502 14:04:21.400065 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-1-root
uuid: "54cf04c1f8cb4ae8918bd36f3b9a21c5"
format_stamp: "Formatted at 2024-05-02 14:04:21 on dist-test-slave-j72s"
I20240502 14:04:21.400368 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:21.415241 11368 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.9.125" port: 33259 } has no permanent_uuid. Determining permanent_uuid...
W20240502 14:04:21.417270 11318 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.9.125:33259: connect: Connection refused (error 111)
W20240502 14:04:21.420842 11368 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.11.9.125:33259: Network error: Client connection negotiation failed: client connection to 127.11.9.125:33259: connect: Connection refused (error 111)
I20240502 14:04:21.426522 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:21.427703 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:21.470585 11368 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.9.125" port: 33259 } attempt: 1
W20240502 14:04:21.474556 11368 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.11.9.125:33259: Network error: Client connection negotiation failed: client connection to 127.11.9.125:33259: connect: Connection refused (error 111)
I20240502 14:04:21.495191 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.125:33259
I20240502 14:04:21.495366 11432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.125:33259 every 8 connection(s)
I20240502 14:04:21.499985 11433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:21.500851 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 14:04:21.505205 11433 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.9.126" port: 41133 } has no permanent_uuid. Determining permanent_uuid...
W20240502 14:04:21.508234 11435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:21.510350 11436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:21.512684 11439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:21.514006 11301 server_base.cc:1034] running on GCE node
I20240502 14:04:21.514889 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:04:21.515146 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:04:21.515341 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658661515326 us; error 0 us; skew 500 ppm
I20240502 14:04:21.515983 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:04:21.520013 11301 webserver.cc:460] Webserver started at http://127.11.9.124:46195/ using document root <none> and password file <none>
I20240502 14:04:21.520249 11433 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.9.125" port: 33259 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:04:21.520489 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:21.520727 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:21.520985 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:21.522625 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-2-root/instance:
uuid: "5028a5be34d84374801e9395ba878ee0"
format_stamp: "Formatted at 2024-05-02 14:04:21 on dist-test-slave-j72s"
I20240502 14:04:21.528070 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240502 14:04:21.531726 11433 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:04:21.533044 11445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:21.535576 11301 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20240502 14:04:21.535928 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-2-root
uuid: "5028a5be34d84374801e9395ba878ee0"
format_stamp: "Formatted at 2024-05-02 14:04:21 on dist-test-slave-j72s"
I20240502 14:04:21.536284 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240502 14:04:21.536999 11433 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.11.9.124:39325: Network error: Client connection negotiation failed: client connection to 127.11.9.124:39325: connect: Connection refused (error 111)
I20240502 14:04:21.542912 11368 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.9.125" port: 33259 } attempt: 2
I20240502 14:04:21.551525 11368 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 } has no permanent_uuid. Determining permanent_uuid...
W20240502 14:04:21.555300 11368 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.11.9.124:39325: Network error: Client connection negotiation failed: client connection to 127.11.9.124:39325: connect: Connection refused (error 111)
I20240502 14:04:21.570501 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:21.571571 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:21.580507 11433 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 } attempt: 1
W20240502 14:04:21.584261 11433 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.11.9.124:39325: Network error: Client connection negotiation failed: client connection to 127.11.9.124:39325: connect: Connection refused (error 111)
I20240502 14:04:21.586751 11368 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 } attempt: 1
W20240502 14:04:21.590196 11368 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.11.9.124:39325: Network error: Client connection negotiation failed: client connection to 127.11.9.124:39325: connect: Connection refused (error 111)
I20240502 14:04:21.635902 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.124:39325
I20240502 14:04:21.636341 11497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.124:39325 every 8 connection(s)
I20240502 14:04:21.639708 11301 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:04:21.640448 11498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:21.645123 11498 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.9.126" port: 41133 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:04:21.655223 11498 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.9.125" port: 33259 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:04:21.657588 11368 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 } attempt: 2
I20240502 14:04:21.660248 11433 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 } attempt: 2
I20240502 14:04:21.667407 11498 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:04:21.681068 11368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261: Bootstrap starting.
I20240502 14:04:21.681896 11433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54cf04c1f8cb4ae8918bd36f3b9a21c5: Bootstrap starting.
I20240502 14:04:21.686581 11368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:21.687995 11433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54cf04c1f8cb4ae8918bd36f3b9a21c5: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:21.688757 11368 log.cc:824] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261: Log is configured to *not* fsync() on all Append() calls
I20240502 14:04:21.688992 11498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5028a5be34d84374801e9395ba878ee0: Bootstrap starting.
I20240502 14:04:21.694854 11498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5028a5be34d84374801e9395ba878ee0: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:21.695140 11433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54cf04c1f8cb4ae8918bd36f3b9a21c5: No bootstrap required, opened a new log
I20240502 14:04:21.695272 11368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261: No bootstrap required, opened a new log
I20240502 14:04:21.700428 11498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5028a5be34d84374801e9395ba878ee0: No bootstrap required, opened a new log
I20240502 14:04:21.715695 11433 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54cf04c1f8cb4ae8918bd36f3b9a21c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115381676a2a4976a81a8a78fd29a261" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 41133 } } peers { permanent_uuid: "54cf04c1f8cb4ae8918bd36f3b9a21c5" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 33259 } } peers { permanent_uuid: "5028a5be34d84374801e9395ba878ee0" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 } }
I20240502 14:04:21.715612 11368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115381676a2a4976a81a8a78fd29a261" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 41133 } } peers { permanent_uuid: "54cf04c1f8cb4ae8918bd36f3b9a21c5" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 33259 } } peers { permanent_uuid: "5028a5be34d84374801e9395ba878ee0" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 } }
I20240502 14:04:21.715612 11498 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5028a5be34d84374801e9395ba878ee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115381676a2a4976a81a8a78fd29a261" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 41133 } } peers { permanent_uuid: "54cf04c1f8cb4ae8918bd36f3b9a21c5" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 33259 } } peers { permanent_uuid: "5028a5be34d84374801e9395ba878ee0" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 } }
I20240502 14:04:21.716509 11433 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54cf04c1f8cb4ae8918bd36f3b9a21c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:21.716604 11498 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5028a5be34d84374801e9395ba878ee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:21.716584 11368 raft_consensus.cc:374] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:21.716832 11433 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54cf04c1f8cb4ae8918bd36f3b9a21c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54cf04c1f8cb4ae8918bd36f3b9a21c5, State: Initialized, Role: FOLLOWER
I20240502 14:04:21.717010 11498 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5028a5be34d84374801e9395ba878ee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5028a5be34d84374801e9395ba878ee0, State: Initialized, Role: FOLLOWER
I20240502 14:04:21.717059 11368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 115381676a2a4976a81a8a78fd29a261, State: Initialized, Role: FOLLOWER
I20240502 14:04:21.717749 11433 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54cf04c1f8cb4ae8918bd36f3b9a21c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115381676a2a4976a81a8a78fd29a261" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 41133 } } peers { permanent_uuid: "54cf04c1f8cb4ae8918bd36f3b9a21c5" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 33259 } } peers { permanent_uuid: "5028a5be34d84374801e9395ba878ee0" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 } }
I20240502 14:04:21.717860 11368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115381676a2a4976a81a8a78fd29a261" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 41133 } } peers { permanent_uuid: "54cf04c1f8cb4ae8918bd36f3b9a21c5" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 33259 } } peers { permanent_uuid: "5028a5be34d84374801e9395ba878ee0" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 } }
I20240502 14:04:21.717862 11498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5028a5be34d84374801e9395ba878ee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115381676a2a4976a81a8a78fd29a261" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 41133 } } peers { permanent_uuid: "54cf04c1f8cb4ae8918bd36f3b9a21c5" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 33259 } } peers { permanent_uuid: "5028a5be34d84374801e9395ba878ee0" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 } }
I20240502 14:04:21.721784 11507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54cf04c1f8cb4ae8918bd36f3b9a21c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115381676a2a4976a81a8a78fd29a261" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 41133 } } peers { permanent_uuid: "54cf04c1f8cb4ae8918bd36f3b9a21c5" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 33259 } } peers { permanent_uuid: "5028a5be34d84374801e9395ba878ee0" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 } } }
I20240502 14:04:21.722728 11507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54cf04c1f8cb4ae8918bd36f3b9a21c5 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:04:21.722539 11509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115381676a2a4976a81a8a78fd29a261" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 41133 } } peers { permanent_uuid: "54cf04c1f8cb4ae8918bd36f3b9a21c5" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 33259 } } peers { permanent_uuid: "5028a5be34d84374801e9395ba878ee0" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 } } }
I20240502 14:04:21.723325 11509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:04:21.723769 11433 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54cf04c1f8cb4ae8918bd36f3b9a21c5 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:04:21.724093 11368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:04:21.724714 11508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5028a5be34d84374801e9395ba878ee0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115381676a2a4976a81a8a78fd29a261" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 41133 } } peers { permanent_uuid: "54cf04c1f8cb4ae8918bd36f3b9a21c5" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 33259 } } peers { permanent_uuid: "5028a5be34d84374801e9395ba878ee0" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 } } }
I20240502 14:04:21.725561 11508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5028a5be34d84374801e9395ba878ee0 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:04:21.727828 11498 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5028a5be34d84374801e9395ba878ee0 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:04:21.778263 11301 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240502 14:04:21.778563 11301 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240502 14:04:21.780458 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:21.782961 11542 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5028a5be34d84374801e9395ba878ee0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 14:04:21.782958 11541 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 14:04:21.783344 11542 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 14:04:21.783397 11541 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 14:04:21.787941 11539 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 54cf04c1f8cb4ae8918bd36f3b9a21c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 14:04:21.788245 11539 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 14:04:21.789444 11544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:21.792167 11545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:21.795074 11547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:21.796151 11301 server_base.cc:1034] running on GCE node
I20240502 14:04:21.797071 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:04:21.797348 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:04:21.797545 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658661797528 us; error 0 us; skew 500 ppm
I20240502 14:04:21.798211 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:04:21.800956 11301 webserver.cc:460] Webserver started at http://127.11.9.65:37853/ using document root <none> and password file <none>
W20240502 14:04:21.800962 11322 tablet.cc:2369] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:04:21.801796 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:21.802067 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:21.802393 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:21.803808 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-0-root/instance:
uuid: "889624e318ae4b6f8483b4dee6ce0ffc"
format_stamp: "Formatted at 2024-05-02 14:04:21 on dist-test-slave-j72s"
I20240502 14:04:21.809690 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20240502 14:04:21.814181 11552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:21.815027 11301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:04:21.815353 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-0-root
uuid: "889624e318ae4b6f8483b4dee6ce0ffc"
format_stamp: "Formatted at 2024-05-02 14:04:21 on dist-test-slave-j72s"
I20240502 14:04:21.815698 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:21.828307 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:21.829617 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:21.831504 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:04:21.835572 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:21.835741 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:21.835979 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:21.836120 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:21.906184 11509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:21.906998 11509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115381676a2a4976a81a8a78fd29a261" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 41133 } } peers { permanent_uuid: "54cf04c1f8cb4ae8918bd36f3b9a21c5" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 33259 } } peers { permanent_uuid: "5028a5be34d84374801e9395ba878ee0" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 } }
I20240502 14:04:21.910849 11408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "115381676a2a4976a81a8a78fd29a261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54cf04c1f8cb4ae8918bd36f3b9a21c5" is_pre_election: true
I20240502 14:04:21.911394 11473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "115381676a2a4976a81a8a78fd29a261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5028a5be34d84374801e9395ba878ee0" is_pre_election: true
I20240502 14:04:21.911747 11408 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 54cf04c1f8cb4ae8918bd36f3b9a21c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 115381676a2a4976a81a8a78fd29a261 in term 0.
I20240502 14:04:21.912356 11473 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5028a5be34d84374801e9395ba878ee0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 115381676a2a4976a81a8a78fd29a261 in term 0.
I20240502 14:04:21.913311 11318 leader_election.cc:302] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 115381676a2a4976a81a8a78fd29a261, 54cf04c1f8cb4ae8918bd36f3b9a21c5; no voters:
I20240502 14:04:21.914587 11509 leader_election.cc:288] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54cf04c1f8cb4ae8918bd36f3b9a21c5 (127.11.9.125:33259), 5028a5be34d84374801e9395ba878ee0 (127.11.9.124:39325)
I20240502 14:04:21.915227 11509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:21.915635 11509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:21.916054 11509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:21.925271 11509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115381676a2a4976a81a8a78fd29a261" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 41133 } } peers { permanent_uuid: "54cf04c1f8cb4ae8918bd36f3b9a21c5" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 33259 } } peers { permanent_uuid: "5028a5be34d84374801e9395ba878ee0" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 } }
I20240502 14:04:21.928190 11408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "115381676a2a4976a81a8a78fd29a261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54cf04c1f8cb4ae8918bd36f3b9a21c5"
I20240502 14:04:21.928787 11408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54cf04c1f8cb4ae8918bd36f3b9a21c5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:21.929468 11473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "115381676a2a4976a81a8a78fd29a261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5028a5be34d84374801e9395ba878ee0"
I20240502 14:04:21.930199 11473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5028a5be34d84374801e9395ba878ee0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:21.935357 11408 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 54cf04c1f8cb4ae8918bd36f3b9a21c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 115381676a2a4976a81a8a78fd29a261 in term 1.
I20240502 14:04:21.936465 11318 leader_election.cc:302] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 115381676a2a4976a81a8a78fd29a261, 54cf04c1f8cb4ae8918bd36f3b9a21c5; no voters:
I20240502 14:04:21.937350 11602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:21.939181 11473 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5028a5be34d84374801e9395ba878ee0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 115381676a2a4976a81a8a78fd29a261 in term 1.
I20240502 14:04:21.940606 11509 leader_election.cc:288] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [CANDIDATE]: Term 1 election: Requested vote from peers 54cf04c1f8cb4ae8918bd36f3b9a21c5 (127.11.9.125:33259), 5028a5be34d84374801e9395ba878ee0 (127.11.9.124:39325)
I20240502 14:04:21.948670 11602 raft_consensus.cc:686] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [term 1 LEADER]: Becoming Leader. State: Replica: 115381676a2a4976a81a8a78fd29a261, State: Running, Role: LEADER
I20240502 14:04:21.949826 11602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115381676a2a4976a81a8a78fd29a261" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 41133 } } peers { permanent_uuid: "54cf04c1f8cb4ae8918bd36f3b9a21c5" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 33259 } } peers { permanent_uuid: "5028a5be34d84374801e9395ba878ee0" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 } }
I20240502 14:04:21.962767 11509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 115381676a2a4976a81a8a78fd29a261. Latest consensus state: current_term: 1 leader_uuid: "115381676a2a4976a81a8a78fd29a261" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115381676a2a4976a81a8a78fd29a261" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 41133 } } peers { permanent_uuid: "54cf04c1f8cb4ae8918bd36f3b9a21c5" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 33259 } } peers { permanent_uuid: "5028a5be34d84374801e9395ba878ee0" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 } } }
I20240502 14:04:21.963572 11509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [sys.catalog]: This master's current role is: LEADER
I20240502 14:04:21.968142 11615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:04:21.978021 11615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:04:21.979542 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.65:44505
I20240502 14:04:21.979885 11618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.65:44505 every 8 connection(s)
I20240502 14:04:22.011247 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 14:04:22.016166 11473 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5028a5be34d84374801e9395ba878ee0 [term 1 FOLLOWER]: Refusing update from remote peer 115381676a2a4976a81a8a78fd29a261: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:04:22.017504 11408 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 54cf04c1f8cb4ae8918bd36f3b9a21c5 [term 1 FOLLOWER]: Refusing update from remote peer 115381676a2a4976a81a8a78fd29a261: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:04:22.018401 11509 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5028a5be34d84374801e9395ba878ee0" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:22.019738 11602 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54cf04c1f8cb4ae8918bd36f3b9a21c5" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 33259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240502 14:04:22.028208 11627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:22.030543 11619 heartbeater.cc:347] Connected to a master server at 127.11.9.124:39325
I20240502 14:04:22.030993 11619 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:22.032363 11619 heartbeater.cc:511] Master 127.11.9.124:39325 requested a full tablet report, sending...
W20240502 14:04:22.034549 11628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:22.053344 11633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:22.061391 11620 heartbeater.cc:347] Connected to a master server at 127.11.9.125:33259
I20240502 14:04:22.061908 11620 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:22.063305 11620 heartbeater.cc:511] Master 127.11.9.125:33259 requested a full tablet report, sending...
I20240502 14:04:22.070706 11507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54cf04c1f8cb4ae8918bd36f3b9a21c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 115381676a2a4976a81a8a78fd29a261. Latest consensus state: current_term: 1 leader_uuid: "115381676a2a4976a81a8a78fd29a261" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115381676a2a4976a81a8a78fd29a261" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 41133 } } peers { permanent_uuid: "54cf04c1f8cb4ae8918bd36f3b9a21c5" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 33259 } } peers { permanent_uuid: "5028a5be34d84374801e9395ba878ee0" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 } } }
I20240502 14:04:22.071441 11507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54cf04c1f8cb4ae8918bd36f3b9a21c5 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:04:22.073105 11622 mvcc.cc:205] Tried to move back new op lower bound from 7023241879567679488 to 7023241879357767680. Current Snapshot: MvccSnapshot[applied={T|T < 7023241879567679488}]
I20240502 14:04:22.077693 11398 ts_manager.cc:194] Registered new tserver with Master: 889624e318ae4b6f8483b4dee6ce0ffc (127.11.9.65:44505)
I20240502 14:04:22.083366 11509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "115381676a2a4976a81a8a78fd29a261" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115381676a2a4976a81a8a78fd29a261" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 41133 } } peers { permanent_uuid: "54cf04c1f8cb4ae8918bd36f3b9a21c5" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 33259 } } peers { permanent_uuid: "5028a5be34d84374801e9395ba878ee0" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 } } }
I20240502 14:04:22.085650 11509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [sys.catalog]: This master's current role is: LEADER
I20240502 14:04:22.086396 11624 heartbeater.cc:347] Connected to a master server at 127.11.9.126:41133
I20240502 14:04:22.086796 11624 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:22.087627 11624 heartbeater.cc:511] Master 127.11.9.126:41133 requested a full tablet report, sending...
I20240502 14:04:22.088173 11463 ts_manager.cc:194] Registered new tserver with Master: 889624e318ae4b6f8483b4dee6ce0ffc (127.11.9.65:44505)
I20240502 14:04:22.089056 11508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5028a5be34d84374801e9395ba878ee0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 115381676a2a4976a81a8a78fd29a261. Latest consensus state: current_term: 1 leader_uuid: "115381676a2a4976a81a8a78fd29a261" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115381676a2a4976a81a8a78fd29a261" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 41133 } } peers { permanent_uuid: "54cf04c1f8cb4ae8918bd36f3b9a21c5" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 33259 } } peers { permanent_uuid: "5028a5be34d84374801e9395ba878ee0" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 } } }
I20240502 14:04:22.089854 11508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5028a5be34d84374801e9395ba878ee0 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:04:22.090301 11301 server_base.cc:1034] running on GCE node
I20240502 14:04:22.091400 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:04:22.091658 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:04:22.091863 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658662091843 us; error 0 us; skew 500 ppm
I20240502 14:04:22.092564 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:04:22.093348 11615 catalog_manager.cc:1347] Generated new cluster ID: d07c2701f63f4270a5b770cb156e7df7
I20240502 14:04:22.093591 11615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:04:22.098948 11333 ts_manager.cc:194] Registered new tserver with Master: 889624e318ae4b6f8483b4dee6ce0ffc (127.11.9.65:44505)
I20240502 14:04:22.104745 11509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "115381676a2a4976a81a8a78fd29a261" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115381676a2a4976a81a8a78fd29a261" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 41133 } } peers { permanent_uuid: "54cf04c1f8cb4ae8918bd36f3b9a21c5" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 33259 } } peers { permanent_uuid: "5028a5be34d84374801e9395ba878ee0" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 } } }
I20240502 14:04:22.106444 11509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [sys.catalog]: This master's current role is: LEADER
I20240502 14:04:22.106400 11507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54cf04c1f8cb4ae8918bd36f3b9a21c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "115381676a2a4976a81a8a78fd29a261" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115381676a2a4976a81a8a78fd29a261" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 41133 } } peers { permanent_uuid: "54cf04c1f8cb4ae8918bd36f3b9a21c5" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 33259 } } peers { permanent_uuid: "5028a5be34d84374801e9395ba878ee0" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 } } }
I20240502 14:04:22.107197 11507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54cf04c1f8cb4ae8918bd36f3b9a21c5 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:04:22.112565 11508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5028a5be34d84374801e9395ba878ee0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "115381676a2a4976a81a8a78fd29a261" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "115381676a2a4976a81a8a78fd29a261" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 41133 } } peers { permanent_uuid: "54cf04c1f8cb4ae8918bd36f3b9a21c5" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 33259 } } peers { permanent_uuid: "5028a5be34d84374801e9395ba878ee0" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 39325 } } }
I20240502 14:04:22.113404 11508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5028a5be34d84374801e9395ba878ee0 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:04:22.114674 11301 webserver.cc:460] Webserver started at http://127.11.9.66:42845/ using document root <none> and password file <none>
I20240502 14:04:22.115312 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:22.115535 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:22.115865 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:22.119704 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-1-root/instance:
uuid: "8b0057d557a143e9a555fb227de8ac2c"
format_stamp: "Formatted at 2024-05-02 14:04:22 on dist-test-slave-j72s"
I20240502 14:04:22.129024 11615 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:04:22.131532 11615 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:04:22.134948 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.005s sys 0.000s
I20240502 14:04:22.139657 11642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:22.140566 11301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20240502 14:04:22.140869 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-1-root
uuid: "8b0057d557a143e9a555fb227de8ac2c"
format_stamp: "Formatted at 2024-05-02 14:04:22 on dist-test-slave-j72s"
I20240502 14:04:22.141181 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:22.158840 11615 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261: Generated new TSK 0
I20240502 14:04:22.163517 11615 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:04:22.164527 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:22.166167 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:22.168699 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:04:22.171447 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:22.171643 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:22.171862 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:22.172053 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:22.263993 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.66:39813
I20240502 14:04:22.264150 11704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.66:39813 every 8 connection(s)
I20240502 14:04:22.282661 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 14:04:22.284492 11705 heartbeater.cc:347] Connected to a master server at 127.11.9.124:39325
I20240502 14:04:22.284890 11705 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:22.285784 11705 heartbeater.cc:511] Master 127.11.9.124:39325 requested a full tablet report, sending...
I20240502 14:04:22.288458 11463 ts_manager.cc:194] Registered new tserver with Master: 8b0057d557a143e9a555fb227de8ac2c (127.11.9.66:39813)
W20240502 14:04:22.298871 11712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:22.300382 11713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:22.301678 11707 heartbeater.cc:347] Connected to a master server at 127.11.9.126:41133
I20240502 14:04:22.302068 11707 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:22.302894 11707 heartbeater.cc:511] Master 127.11.9.126:41133 requested a full tablet report, sending...
I20240502 14:04:22.305071 11333 ts_manager.cc:194] Registered new tserver with Master: 8b0057d557a143e9a555fb227de8ac2c (127.11.9.66:39813)
I20240502 14:04:22.308557 11301 server_base.cc:1034] running on GCE node
I20240502 14:04:22.308030 11333 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47340
W20240502 14:04:22.310775 11715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:22.311748 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 14:04:22.311790 11706 heartbeater.cc:347] Connected to a master server at 127.11.9.125:33259
I20240502 14:04:22.312335 11706 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:22.313011 11706 heartbeater.cc:511] Master 127.11.9.125:33259 requested a full tablet report, sending...
W20240502 14:04:22.313647 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:04:22.313859 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658662313838 us; error 0 us; skew 500 ppm
I20240502 14:04:22.314392 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:04:22.315605 11398 ts_manager.cc:194] Registered new tserver with Master: 8b0057d557a143e9a555fb227de8ac2c (127.11.9.66:39813)
I20240502 14:04:22.317355 11301 webserver.cc:460] Webserver started at http://127.11.9.67:39075/ using document root <none> and password file <none>
I20240502 14:04:22.317890 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:22.318084 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:22.318346 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:22.319324 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-2-root/instance:
uuid: "72fdba6cc72144fbae68a9a77616d6b9"
format_stamp: "Formatted at 2024-05-02 14:04:22 on dist-test-slave-j72s"
I20240502 14:04:22.323518 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:04:22.327122 11720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:22.328074 11301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20240502 14:04:22.328341 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-2-root
uuid: "72fdba6cc72144fbae68a9a77616d6b9"
format_stamp: "Formatted at 2024-05-02 14:04:22 on dist-test-slave-j72s"
I20240502 14:04:22.328689 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:22.394657 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:22.395900 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:22.398329 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:04:22.401437 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:22.401660 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:22.401928 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:22.402065 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:22.499118 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.67:43747
I20240502 14:04:22.499485 11782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.67:43747 every 8 connection(s)
I20240502 14:04:22.525697 11783 heartbeater.cc:347] Connected to a master server at 127.11.9.124:39325
I20240502 14:04:22.526185 11783 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:22.527454 11783 heartbeater.cc:511] Master 127.11.9.124:39325 requested a full tablet report, sending...
I20240502 14:04:22.530514 11463 ts_manager.cc:194] Registered new tserver with Master: 72fdba6cc72144fbae68a9a77616d6b9 (127.11.9.67:43747)
I20240502 14:04:22.534111 11784 heartbeater.cc:347] Connected to a master server at 127.11.9.125:33259
I20240502 14:04:22.534511 11784 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:22.535432 11784 heartbeater.cc:511] Master 127.11.9.125:33259 requested a full tablet report, sending...
I20240502 14:04:22.540151 11398 ts_manager.cc:194] Registered new tserver with Master: 72fdba6cc72144fbae68a9a77616d6b9 (127.11.9.67:43747)
I20240502 14:04:22.540642 11786 heartbeater.cc:347] Connected to a master server at 127.11.9.126:41133
I20240502 14:04:22.541038 11786 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:22.541936 11786 heartbeater.cc:511] Master 127.11.9.126:41133 requested a full tablet report, sending...
I20240502 14:04:22.543917 11333 ts_manager.cc:194] Registered new tserver with Master: 72fdba6cc72144fbae68a9a77616d6b9 (127.11.9.67:43747)
I20240502 14:04:22.544042 11301 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.021884814s
I20240502 14:04:22.545778 11333 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47356
I20240502 14:04:22.788789 11542 catalog_manager.cc:1259] Loaded cluster ID: d07c2701f63f4270a5b770cb156e7df7
I20240502 14:04:22.789127 11542 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5028a5be34d84374801e9395ba878ee0: loading cluster ID for follower catalog manager: success
I20240502 14:04:22.791651 11539 catalog_manager.cc:1259] Loaded cluster ID: d07c2701f63f4270a5b770cb156e7df7
I20240502 14:04:22.791931 11539 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 54cf04c1f8cb4ae8918bd36f3b9a21c5: loading cluster ID for follower catalog manager: success
I20240502 14:04:22.795620 11542 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5028a5be34d84374801e9395ba878ee0: acquiring CA information for follower catalog manager: success
I20240502 14:04:22.796129 11539 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 54cf04c1f8cb4ae8918bd36f3b9a21c5: acquiring CA information for follower catalog manager: success
I20240502 14:04:22.799233 11539 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 54cf04c1f8cb4ae8918bd36f3b9a21c5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 14:04:22.800005 11542 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5028a5be34d84374801e9395ba878ee0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 14:04:23.102929 11333 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47328
I20240502 14:04:23.312289 11707 heartbeater.cc:503] Master 127.11.9.126:41133 was elected leader, sending a full tablet report...
I20240502 14:04:23.549206 11786 heartbeater.cc:503] Master 127.11.9.126:41133 was elected leader, sending a full tablet report...
I20240502 14:04:23.774499 11534 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:04:23.837992 11301 test_util.cc:268] Using random seed: 345092366
I20240502 14:04:23.891685 11333 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47368:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 14:04:23.964735 11670 tablet_service.cc:1467] Processing CreateTablet for tablet 0e45e86559034240a302c4809c274be0 (DEFAULT_TABLE table=test-workload [id=4cff8ec93f564bd98f0a52a899b270cc]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 14:04:23.966814 11670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e45e86559034240a302c4809c274be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:23.966630 11580 tablet_service.cc:1467] Processing CreateTablet for tablet 6474911baea94b3ca24882df75f78db3 (DEFAULT_TABLE table=test-workload [id=4cff8ec93f564bd98f0a52a899b270cc]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 14:04:23.968391 11580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6474911baea94b3ca24882df75f78db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:23.968562 11579 tablet_service.cc:1467] Processing CreateTablet for tablet b4b68832dc034a619f77c34ccff2e3e9 (DEFAULT_TABLE table=test-workload [id=4cff8ec93f564bd98f0a52a899b270cc]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 14:04:23.979084 11579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b68832dc034a619f77c34ccff2e3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:23.998904 11819 tablet_bootstrap.cc:492] T 6474911baea94b3ca24882df75f78db3 P 889624e318ae4b6f8483b4dee6ce0ffc: Bootstrap starting.
I20240502 14:04:24.004751 11819 tablet_bootstrap.cc:654] T 6474911baea94b3ca24882df75f78db3 P 889624e318ae4b6f8483b4dee6ce0ffc: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:24.007030 11748 tablet_service.cc:1467] Processing CreateTablet for tablet 3103c6d9f38945fe9d045ab485238461 (DEFAULT_TABLE table=test-workload [id=4cff8ec93f564bd98f0a52a899b270cc]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 14:04:24.008379 11748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3103c6d9f38945fe9d045ab485238461. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:24.013600 11821 tablet_bootstrap.cc:492] T 0e45e86559034240a302c4809c274be0 P 8b0057d557a143e9a555fb227de8ac2c: Bootstrap starting.
I20240502 14:04:24.014741 11819 tablet_bootstrap.cc:492] T 6474911baea94b3ca24882df75f78db3 P 889624e318ae4b6f8483b4dee6ce0ffc: No bootstrap required, opened a new log
I20240502 14:04:24.015230 11819 ts_tablet_manager.cc:1367] T 6474911baea94b3ca24882df75f78db3 P 889624e318ae4b6f8483b4dee6ce0ffc: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.007s
I20240502 14:04:24.018358 11819 raft_consensus.cc:348] T 6474911baea94b3ca24882df75f78db3 P 889624e318ae4b6f8483b4dee6ce0ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889624e318ae4b6f8483b4dee6ce0ffc" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44505 } }
I20240502 14:04:24.018934 11819 raft_consensus.cc:374] T 6474911baea94b3ca24882df75f78db3 P 889624e318ae4b6f8483b4dee6ce0ffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:24.019237 11819 raft_consensus.cc:729] T 6474911baea94b3ca24882df75f78db3 P 889624e318ae4b6f8483b4dee6ce0ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889624e318ae4b6f8483b4dee6ce0ffc, State: Initialized, Role: FOLLOWER
I20240502 14:04:24.020107 11821 tablet_bootstrap.cc:654] T 0e45e86559034240a302c4809c274be0 P 8b0057d557a143e9a555fb227de8ac2c: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:24.019889 11819 consensus_queue.cc:260] T 6474911baea94b3ca24882df75f78db3 P 889624e318ae4b6f8483b4dee6ce0ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889624e318ae4b6f8483b4dee6ce0ffc" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44505 } }
I20240502 14:04:24.020596 11819 raft_consensus.cc:388] T 6474911baea94b3ca24882df75f78db3 P 889624e318ae4b6f8483b4dee6ce0ffc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:04:24.020953 11819 raft_consensus.cc:482] T 6474911baea94b3ca24882df75f78db3 P 889624e318ae4b6f8483b4dee6ce0ffc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:04:24.021313 11819 raft_consensus.cc:3037] T 6474911baea94b3ca24882df75f78db3 P 889624e318ae4b6f8483b4dee6ce0ffc [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:24.024549 11822 tablet_bootstrap.cc:492] T 3103c6d9f38945fe9d045ab485238461 P 72fdba6cc72144fbae68a9a77616d6b9: Bootstrap starting.
I20240502 14:04:24.026764 11821 tablet_bootstrap.cc:492] T 0e45e86559034240a302c4809c274be0 P 8b0057d557a143e9a555fb227de8ac2c: No bootstrap required, opened a new log
I20240502 14:04:24.027186 11821 ts_tablet_manager.cc:1367] T 0e45e86559034240a302c4809c274be0 P 8b0057d557a143e9a555fb227de8ac2c: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20240502 14:04:24.027634 11819 raft_consensus.cc:504] T 6474911baea94b3ca24882df75f78db3 P 889624e318ae4b6f8483b4dee6ce0ffc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889624e318ae4b6f8483b4dee6ce0ffc" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44505 } }
I20240502 14:04:24.028390 11819 leader_election.cc:302] T 6474911baea94b3ca24882df75f78db3 P 889624e318ae4b6f8483b4dee6ce0ffc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 889624e318ae4b6f8483b4dee6ce0ffc; no voters:
I20240502 14:04:24.030079 11821 raft_consensus.cc:348] T 0e45e86559034240a302c4809c274be0 P 8b0057d557a143e9a555fb227de8ac2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0057d557a143e9a555fb227de8ac2c" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 39813 } }
I20240502 14:04:24.030776 11822 tablet_bootstrap.cc:654] T 3103c6d9f38945fe9d045ab485238461 P 72fdba6cc72144fbae68a9a77616d6b9: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:24.030751 11821 raft_consensus.cc:374] T 0e45e86559034240a302c4809c274be0 P 8b0057d557a143e9a555fb227de8ac2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:24.031370 11821 raft_consensus.cc:729] T 0e45e86559034240a302c4809c274be0 P 8b0057d557a143e9a555fb227de8ac2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0057d557a143e9a555fb227de8ac2c, State: Initialized, Role: FOLLOWER
I20240502 14:04:24.032243 11821 consensus_queue.cc:260] T 0e45e86559034240a302c4809c274be0 P 8b0057d557a143e9a555fb227de8ac2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0057d557a143e9a555fb227de8ac2c" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 39813 } }
I20240502 14:04:24.033727 11821 raft_consensus.cc:388] T 0e45e86559034240a302c4809c274be0 P 8b0057d557a143e9a555fb227de8ac2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:04:24.034137 11819 leader_election.cc:288] T 6474911baea94b3ca24882df75f78db3 P 889624e318ae4b6f8483b4dee6ce0ffc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:04:24.034128 11821 raft_consensus.cc:482] T 0e45e86559034240a302c4809c274be0 P 8b0057d557a143e9a555fb227de8ac2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:04:24.034792 11821 raft_consensus.cc:3037] T 0e45e86559034240a302c4809c274be0 P 8b0057d557a143e9a555fb227de8ac2c [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:24.034921 11824 raft_consensus.cc:2781] T 6474911baea94b3ca24882df75f78db3 P 889624e318ae4b6f8483b4dee6ce0ffc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:24.042812 11822 tablet_bootstrap.cc:492] T 3103c6d9f38945fe9d045ab485238461 P 72fdba6cc72144fbae68a9a77616d6b9: No bootstrap required, opened a new log
I20240502 14:04:24.043356 11822 ts_tablet_manager.cc:1367] T 3103c6d9f38945fe9d045ab485238461 P 72fdba6cc72144fbae68a9a77616d6b9: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.013s
I20240502 14:04:24.043445 11821 raft_consensus.cc:504] T 0e45e86559034240a302c4809c274be0 P 8b0057d557a143e9a555fb227de8ac2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0057d557a143e9a555fb227de8ac2c" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 39813 } }
I20240502 14:04:24.044624 11821 leader_election.cc:302] T 0e45e86559034240a302c4809c274be0 P 8b0057d557a143e9a555fb227de8ac2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b0057d557a143e9a555fb227de8ac2c; no voters:
I20240502 14:04:24.046782 11822 raft_consensus.cc:348] T 3103c6d9f38945fe9d045ab485238461 P 72fdba6cc72144fbae68a9a77616d6b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72fdba6cc72144fbae68a9a77616d6b9" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 43747 } }
I20240502 14:04:24.048857 11819 ts_tablet_manager.cc:1398] T 6474911baea94b3ca24882df75f78db3 P 889624e318ae4b6f8483b4dee6ce0ffc: Time spent starting tablet: real 0.033s user 0.014s sys 0.018s
I20240502 14:04:24.047457 11822 raft_consensus.cc:374] T 3103c6d9f38945fe9d045ab485238461 P 72fdba6cc72144fbae68a9a77616d6b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:24.050962 11822 raft_consensus.cc:729] T 3103c6d9f38945fe9d045ab485238461 P 72fdba6cc72144fbae68a9a77616d6b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72fdba6cc72144fbae68a9a77616d6b9, State: Initialized, Role: FOLLOWER
I20240502 14:04:24.051481 11624 heartbeater.cc:503] Master 127.11.9.126:41133 was elected leader, sending a full tablet report...
I20240502 14:04:24.051820 11822 consensus_queue.cc:260] T 3103c6d9f38945fe9d045ab485238461 P 72fdba6cc72144fbae68a9a77616d6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72fdba6cc72144fbae68a9a77616d6b9" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 43747 } }
I20240502 14:04:24.052801 11822 raft_consensus.cc:388] T 3103c6d9f38945fe9d045ab485238461 P 72fdba6cc72144fbae68a9a77616d6b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:04:24.053189 11822 raft_consensus.cc:482] T 3103c6d9f38945fe9d045ab485238461 P 72fdba6cc72144fbae68a9a77616d6b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:04:24.053354 11819 tablet_bootstrap.cc:492] T b4b68832dc034a619f77c34ccff2e3e9 P 889624e318ae4b6f8483b4dee6ce0ffc: Bootstrap starting.
I20240502 14:04:24.053668 11822 raft_consensus.cc:3037] T 3103c6d9f38945fe9d045ab485238461 P 72fdba6cc72144fbae68a9a77616d6b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:24.058887 11824 raft_consensus.cc:686] T 6474911baea94b3ca24882df75f78db3 P 889624e318ae4b6f8483b4dee6ce0ffc [term 1 LEADER]: Becoming Leader. State: Replica: 889624e318ae4b6f8483b4dee6ce0ffc, State: Running, Role: LEADER
I20240502 14:04:24.059769 11824 consensus_queue.cc:237] T 6474911baea94b3ca24882df75f78db3 P 889624e318ae4b6f8483b4dee6ce0ffc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889624e318ae4b6f8483b4dee6ce0ffc" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44505 } }
I20240502 14:04:24.061830 11821 leader_election.cc:288] T 0e45e86559034240a302c4809c274be0 P 8b0057d557a143e9a555fb227de8ac2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:04:24.062601 11822 raft_consensus.cc:504] T 3103c6d9f38945fe9d045ab485238461 P 72fdba6cc72144fbae68a9a77616d6b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72fdba6cc72144fbae68a9a77616d6b9" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 43747 } }
I20240502 14:04:24.063365 11822 leader_election.cc:302] T 3103c6d9f38945fe9d045ab485238461 P 72fdba6cc72144fbae68a9a77616d6b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72fdba6cc72144fbae68a9a77616d6b9; no voters:
I20240502 14:04:24.063953 11819 tablet_bootstrap.cc:654] T b4b68832dc034a619f77c34ccff2e3e9 P 889624e318ae4b6f8483b4dee6ce0ffc: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:24.066681 11828 raft_consensus.cc:2781] T 0e45e86559034240a302c4809c274be0 P 8b0057d557a143e9a555fb227de8ac2c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:24.080400 11828 raft_consensus.cc:686] T 0e45e86559034240a302c4809c274be0 P 8b0057d557a143e9a555fb227de8ac2c [term 1 LEADER]: Becoming Leader. State: Replica: 8b0057d557a143e9a555fb227de8ac2c, State: Running, Role: LEADER
I20240502 14:04:24.081310 11828 consensus_queue.cc:237] T 0e45e86559034240a302c4809c274be0 P 8b0057d557a143e9a555fb227de8ac2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0057d557a143e9a555fb227de8ac2c" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 39813 } }
I20240502 14:04:24.084780 11822 leader_election.cc:288] T 3103c6d9f38945fe9d045ab485238461 P 72fdba6cc72144fbae68a9a77616d6b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:04:24.088869 11831 raft_consensus.cc:2781] T 3103c6d9f38945fe9d045ab485238461 P 72fdba6cc72144fbae68a9a77616d6b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:24.121142 11333 catalog_manager.cc:5484] T 0e45e86559034240a302c4809c274be0 P 8b0057d557a143e9a555fb227de8ac2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b0057d557a143e9a555fb227de8ac2c (127.11.9.66). New cstate: current_term: 1 leader_uuid: "8b0057d557a143e9a555fb227de8ac2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0057d557a143e9a555fb227de8ac2c" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 39813 } health_report { overall_health: HEALTHY } } }
I20240502 14:04:24.128697 11822 ts_tablet_manager.cc:1398] T 3103c6d9f38945fe9d045ab485238461 P 72fdba6cc72144fbae68a9a77616d6b9: Time spent starting tablet: real 0.085s user 0.029s sys 0.030s
I20240502 14:04:24.133208 11821 ts_tablet_manager.cc:1398] T 0e45e86559034240a302c4809c274be0 P 8b0057d557a143e9a555fb227de8ac2c: Time spent starting tablet: real 0.106s user 0.020s sys 0.026s
I20240502 14:04:24.147981 11332 catalog_manager.cc:5484] T 6474911baea94b3ca24882df75f78db3 P 889624e318ae4b6f8483b4dee6ce0ffc reported cstate change: term changed from 0 to 1, leader changed from <none> to 889624e318ae4b6f8483b4dee6ce0ffc (127.11.9.65). New cstate: current_term: 1 leader_uuid: "889624e318ae4b6f8483b4dee6ce0ffc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889624e318ae4b6f8483b4dee6ce0ffc" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44505 } health_report { overall_health: HEALTHY } } }
I20240502 14:04:24.158066 11819 tablet_bootstrap.cc:492] T b4b68832dc034a619f77c34ccff2e3e9 P 889624e318ae4b6f8483b4dee6ce0ffc: No bootstrap required, opened a new log
I20240502 14:04:24.158529 11819 ts_tablet_manager.cc:1367] T b4b68832dc034a619f77c34ccff2e3e9 P 889624e318ae4b6f8483b4dee6ce0ffc: Time spent bootstrapping tablet: real 0.106s user 0.029s sys 0.035s
I20240502 14:04:24.160907 11819 raft_consensus.cc:348] T b4b68832dc034a619f77c34ccff2e3e9 P 889624e318ae4b6f8483b4dee6ce0ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889624e318ae4b6f8483b4dee6ce0ffc" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44505 } }
I20240502 14:04:24.161482 11819 raft_consensus.cc:374] T b4b68832dc034a619f77c34ccff2e3e9 P 889624e318ae4b6f8483b4dee6ce0ffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:24.161780 11819 raft_consensus.cc:729] T b4b68832dc034a619f77c34ccff2e3e9 P 889624e318ae4b6f8483b4dee6ce0ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889624e318ae4b6f8483b4dee6ce0ffc, State: Initialized, Role: FOLLOWER
I20240502 14:04:24.162361 11819 consensus_queue.cc:260] T b4b68832dc034a619f77c34ccff2e3e9 P 889624e318ae4b6f8483b4dee6ce0ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889624e318ae4b6f8483b4dee6ce0ffc" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44505 } }
I20240502 14:04:24.162945 11819 raft_consensus.cc:388] T b4b68832dc034a619f77c34ccff2e3e9 P 889624e318ae4b6f8483b4dee6ce0ffc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:04:24.163230 11819 raft_consensus.cc:482] T b4b68832dc034a619f77c34ccff2e3e9 P 889624e318ae4b6f8483b4dee6ce0ffc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:04:24.163576 11819 raft_consensus.cc:3037] T b4b68832dc034a619f77c34ccff2e3e9 P 889624e318ae4b6f8483b4dee6ce0ffc [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:24.155241 11831 raft_consensus.cc:686] T 3103c6d9f38945fe9d045ab485238461 P 72fdba6cc72144fbae68a9a77616d6b9 [term 1 LEADER]: Becoming Leader. State: Replica: 72fdba6cc72144fbae68a9a77616d6b9, State: Running, Role: LEADER
I20240502 14:04:24.164866 11831 consensus_queue.cc:237] T 3103c6d9f38945fe9d045ab485238461 P 72fdba6cc72144fbae68a9a77616d6b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72fdba6cc72144fbae68a9a77616d6b9" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 43747 } }
I20240502 14:04:24.171645 11819 raft_consensus.cc:504] T b4b68832dc034a619f77c34ccff2e3e9 P 889624e318ae4b6f8483b4dee6ce0ffc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889624e318ae4b6f8483b4dee6ce0ffc" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44505 } }
I20240502 14:04:24.172442 11819 leader_election.cc:302] T b4b68832dc034a619f77c34ccff2e3e9 P 889624e318ae4b6f8483b4dee6ce0ffc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 889624e318ae4b6f8483b4dee6ce0ffc; no voters:
I20240502 14:04:24.173555 11827 raft_consensus.cc:2781] T b4b68832dc034a619f77c34ccff2e3e9 P 889624e318ae4b6f8483b4dee6ce0ffc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:24.174027 11827 raft_consensus.cc:686] T b4b68832dc034a619f77c34ccff2e3e9 P 889624e318ae4b6f8483b4dee6ce0ffc [term 1 LEADER]: Becoming Leader. State: Replica: 889624e318ae4b6f8483b4dee6ce0ffc, State: Running, Role: LEADER
I20240502 14:04:24.173341 11819 leader_election.cc:288] T b4b68832dc034a619f77c34ccff2e3e9 P 889624e318ae4b6f8483b4dee6ce0ffc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:04:24.177318 11819 ts_tablet_manager.cc:1398] T b4b68832dc034a619f77c34ccff2e3e9 P 889624e318ae4b6f8483b4dee6ce0ffc: Time spent starting tablet: real 0.018s user 0.009s sys 0.006s
I20240502 14:04:24.177407 11827 consensus_queue.cc:237] T b4b68832dc034a619f77c34ccff2e3e9 P 889624e318ae4b6f8483b4dee6ce0ffc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889624e318ae4b6f8483b4dee6ce0ffc" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44505 } }
I20240502 14:04:24.184545 11332 catalog_manager.cc:5484] T 3103c6d9f38945fe9d045ab485238461 P 72fdba6cc72144fbae68a9a77616d6b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72fdba6cc72144fbae68a9a77616d6b9 (127.11.9.67). New cstate: current_term: 1 leader_uuid: "72fdba6cc72144fbae68a9a77616d6b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72fdba6cc72144fbae68a9a77616d6b9" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 43747 } health_report { overall_health: HEALTHY } } }
I20240502 14:04:24.206084 11333 catalog_manager.cc:5484] T b4b68832dc034a619f77c34ccff2e3e9 P 889624e318ae4b6f8483b4dee6ce0ffc reported cstate change: term changed from 0 to 1, leader changed from <none> to 889624e318ae4b6f8483b4dee6ce0ffc (127.11.9.65). New cstate: current_term: 1 leader_uuid: "889624e318ae4b6f8483b4dee6ce0ffc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889624e318ae4b6f8483b4dee6ce0ffc" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44505 } health_report { overall_health: HEALTHY } } }
I20240502 14:04:24.259166 11301 tablet_server.cc:178] TabletServer@127.11.9.65:0 shutting down...
I20240502 14:04:24.279896 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:04:24.280695 11301 tablet_replica.cc:330] T b4b68832dc034a619f77c34ccff2e3e9 P 889624e318ae4b6f8483b4dee6ce0ffc: stopping tablet replica
I20240502 14:04:24.281301 11301 raft_consensus.cc:2229] T b4b68832dc034a619f77c34ccff2e3e9 P 889624e318ae4b6f8483b4dee6ce0ffc [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:04:24.281777 11301 raft_consensus.cc:2258] T b4b68832dc034a619f77c34ccff2e3e9 P 889624e318ae4b6f8483b4dee6ce0ffc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:24.284322 11301 tablet_replica.cc:330] T 6474911baea94b3ca24882df75f78db3 P 889624e318ae4b6f8483b4dee6ce0ffc: stopping tablet replica
I20240502 14:04:24.284754 11301 raft_consensus.cc:2229] T 6474911baea94b3ca24882df75f78db3 P 889624e318ae4b6f8483b4dee6ce0ffc [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:04:24.285116 11301 raft_consensus.cc:2258] T 6474911baea94b3ca24882df75f78db3 P 889624e318ae4b6f8483b4dee6ce0ffc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:24.429234 11301 tablet_server.cc:194] TabletServer@127.11.9.65:0 shutdown complete.
I20240502 14:04:24.438088 11301 tablet_server.cc:178] TabletServer@127.11.9.66:0 shutting down...
I20240502 14:04:24.455157 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:04:24.455899 11301 tablet_replica.cc:330] T 0e45e86559034240a302c4809c274be0 P 8b0057d557a143e9a555fb227de8ac2c: stopping tablet replica
I20240502 14:04:24.456400 11301 raft_consensus.cc:2229] T 0e45e86559034240a302c4809c274be0 P 8b0057d557a143e9a555fb227de8ac2c [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:04:24.456772 11301 raft_consensus.cc:2258] T 0e45e86559034240a302c4809c274be0 P 8b0057d557a143e9a555fb227de8ac2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:24.475517 11301 tablet_server.cc:194] TabletServer@127.11.9.66:0 shutdown complete.
I20240502 14:04:24.482303 11301 tablet_server.cc:178] TabletServer@127.11.9.67:0 shutting down...
I20240502 14:04:24.496574 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:04:24.497275 11301 tablet_replica.cc:330] T 3103c6d9f38945fe9d045ab485238461 P 72fdba6cc72144fbae68a9a77616d6b9: stopping tablet replica
I20240502 14:04:24.497817 11301 raft_consensus.cc:2229] T 3103c6d9f38945fe9d045ab485238461 P 72fdba6cc72144fbae68a9a77616d6b9 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:04:24.498208 11301 raft_consensus.cc:2258] T 3103c6d9f38945fe9d045ab485238461 P 72fdba6cc72144fbae68a9a77616d6b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:24.517153 11301 tablet_server.cc:194] TabletServer@127.11.9.67:0 shutdown complete.
I20240502 14:04:24.524029 11301 master.cc:537] Master@127.11.9.126:41133 shutting down...
I20240502 14:04:24.611297 11301 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:04:24.612105 11301 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:24.612432 11301 tablet_replica.cc:330] T 00000000000000000000000000000000 P 115381676a2a4976a81a8a78fd29a261: stopping tablet replica
I20240502 14:04:24.639380 11301 master.cc:552] Master@127.11.9.126:41133 shutdown complete.
I20240502 14:04:24.650954 11301 master.cc:537] Master@127.11.9.125:33259 shutting down...
I20240502 14:04:24.663246 11301 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 54cf04c1f8cb4ae8918bd36f3b9a21c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:24.663650 11301 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 54cf04c1f8cb4ae8918bd36f3b9a21c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:24.663985 11301 tablet_replica.cc:330] T 00000000000000000000000000000000 P 54cf04c1f8cb4ae8918bd36f3b9a21c5: stopping tablet replica
I20240502 14:04:24.709247 11301 master.cc:552] Master@127.11.9.125:33259 shutdown complete.
I20240502 14:04:24.740056 11301 master.cc:537] Master@127.11.9.124:39325 shutting down...
I20240502 14:04:24.752167 11301 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5028a5be34d84374801e9395ba878ee0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:24.752758 11301 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5028a5be34d84374801e9395ba878ee0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:24.753021 11301 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5028a5be34d84374801e9395ba878ee0: stopping tablet replica
I20240502 14:04:24.769799 11301 master.cc:552] Master@127.11.9.124:39325 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (4726 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240502 14:04:24.793112 11301 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.9.126:36455
I20240502 14:04:24.794059 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:24.799160 11842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:24.799672 11843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:24.800235 11845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:24.800701 11301 server_base.cc:1034] running on GCE node
I20240502 14:04:24.801328 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:04:24.801491 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:04:24.801640 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658664801624 us; error 0 us; skew 500 ppm
I20240502 14:04:24.802062 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:04:24.804064 11301 webserver.cc:460] Webserver started at http://127.11.9.126:42309/ using document root <none> and password file <none>
I20240502 14:04:24.804479 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:24.804641 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:24.804883 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:24.805953 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658659912502-11301-0/minicluster-data/master-0-root/instance:
uuid: "11d49c87b2cb4ebb9f6d72955afe2cd0"
format_stamp: "Formatted at 2024-05-02 14:04:24 on dist-test-slave-j72s"
I20240502 14:04:24.810223 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240502 14:04:24.813050 11850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:24.813743 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:04:24.814010 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658659912502-11301-0/minicluster-data/master-0-root
uuid: "11d49c87b2cb4ebb9f6d72955afe2cd0"
format_stamp: "Formatted at 2024-05-02 14:04:24 on dist-test-slave-j72s"
I20240502 14:04:24.814251 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658659912502-11301-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658659912502-11301-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658659912502-11301-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:24.838505 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:24.839339 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:24.870378 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.126:36455
I20240502 14:04:24.870461 11901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.126:36455 every 8 connection(s)
I20240502 14:04:24.873883 11902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:24.883028 11902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11d49c87b2cb4ebb9f6d72955afe2cd0: Bootstrap starting.
I20240502 14:04:24.886469 11902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11d49c87b2cb4ebb9f6d72955afe2cd0: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:24.889886 11902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11d49c87b2cb4ebb9f6d72955afe2cd0: No bootstrap required, opened a new log
I20240502 14:04:24.891465 11902 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11d49c87b2cb4ebb9f6d72955afe2cd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11d49c87b2cb4ebb9f6d72955afe2cd0" member_type: VOTER }
I20240502 14:04:24.891799 11902 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11d49c87b2cb4ebb9f6d72955afe2cd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:24.891999 11902 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11d49c87b2cb4ebb9f6d72955afe2cd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11d49c87b2cb4ebb9f6d72955afe2cd0, State: Initialized, Role: FOLLOWER
I20240502 14:04:24.892439 11902 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11d49c87b2cb4ebb9f6d72955afe2cd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11d49c87b2cb4ebb9f6d72955afe2cd0" member_type: VOTER }
I20240502 14:04:24.892841 11902 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11d49c87b2cb4ebb9f6d72955afe2cd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:04:24.893020 11902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11d49c87b2cb4ebb9f6d72955afe2cd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:04:24.893242 11902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11d49c87b2cb4ebb9f6d72955afe2cd0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:24.897426 11902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11d49c87b2cb4ebb9f6d72955afe2cd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11d49c87b2cb4ebb9f6d72955afe2cd0" member_type: VOTER }
I20240502 14:04:24.897883 11902 leader_election.cc:302] T 00000000000000000000000000000000 P 11d49c87b2cb4ebb9f6d72955afe2cd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11d49c87b2cb4ebb9f6d72955afe2cd0; no voters:
I20240502 14:04:24.898857 11902 leader_election.cc:288] T 00000000000000000000000000000000 P 11d49c87b2cb4ebb9f6d72955afe2cd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:04:24.899232 11905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11d49c87b2cb4ebb9f6d72955afe2cd0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:24.900554 11905 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11d49c87b2cb4ebb9f6d72955afe2cd0 [term 1 LEADER]: Becoming Leader. State: Replica: 11d49c87b2cb4ebb9f6d72955afe2cd0, State: Running, Role: LEADER
I20240502 14:04:24.901175 11905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11d49c87b2cb4ebb9f6d72955afe2cd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11d49c87b2cb4ebb9f6d72955afe2cd0" member_type: VOTER }
I20240502 14:04:24.901459 11902 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11d49c87b2cb4ebb9f6d72955afe2cd0 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:04:24.905594 11907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11d49c87b2cb4ebb9f6d72955afe2cd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11d49c87b2cb4ebb9f6d72955afe2cd0. Latest consensus state: current_term: 1 leader_uuid: "11d49c87b2cb4ebb9f6d72955afe2cd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11d49c87b2cb4ebb9f6d72955afe2cd0" member_type: VOTER } }
I20240502 14:04:24.905707 11906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11d49c87b2cb4ebb9f6d72955afe2cd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11d49c87b2cb4ebb9f6d72955afe2cd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11d49c87b2cb4ebb9f6d72955afe2cd0" member_type: VOTER } }
I20240502 14:04:24.906320 11907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11d49c87b2cb4ebb9f6d72955afe2cd0 [sys.catalog]: This master's current role is: LEADER
I20240502 14:04:24.906464 11906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11d49c87b2cb4ebb9f6d72955afe2cd0 [sys.catalog]: This master's current role is: LEADER
I20240502 14:04:24.912012 11913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:04:24.918872 11913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:04:24.919719 11301 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:04:24.926316 11913 catalog_manager.cc:1347] Generated new cluster ID: 1371396652e844de897d70d68f231981
I20240502 14:04:24.926589 11913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:04:24.955068 11913 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:04:24.956223 11913 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:04:24.967337 11913 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 11d49c87b2cb4ebb9f6d72955afe2cd0: Generated new TSK 0
I20240502 14:04:24.967912 11913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:04:24.985638 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:24.991676 11923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:24.992765 11924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:24.993971 11926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:24.996963 11301 server_base.cc:1034] running on GCE node
I20240502 14:04:24.998296 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:04:24.998490 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:04:24.998659 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658664998628 us; error 0 us; skew 500 ppm
I20240502 14:04:24.999166 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:04:25.001391 11301 webserver.cc:460] Webserver started at http://127.11.9.65:38671/ using document root <none> and password file <none>
I20240502 14:04:25.001843 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:25.002004 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:25.002242 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:25.003309 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658659912502-11301-0/minicluster-data/ts-0-root/instance:
uuid: "5c6d12fd1dd9431e9caac1ca05c3640f"
format_stamp: "Formatted at 2024-05-02 14:04:24 on dist-test-slave-j72s"
I20240502 14:04:25.007393 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20240502 14:04:25.010354 11931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:25.011068 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:04:25.011443 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658659912502-11301-0/minicluster-data/ts-0-root
uuid: "5c6d12fd1dd9431e9caac1ca05c3640f"
format_stamp: "Formatted at 2024-05-02 14:04:24 on dist-test-slave-j72s"
I20240502 14:04:25.011704 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658659912502-11301-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658659912502-11301-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658659912502-11301-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:25.022148 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:25.022994 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:25.024267 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:04:25.026373 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:25.026571 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:25.026818 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:25.026978 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:25.131381 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.65:45197
I20240502 14:04:25.131469 11993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.65:45197 every 8 connection(s)
I20240502 14:04:25.143790 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:25.150872 11996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:25.151710 11997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:25.154630 11999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:25.156513 11301 server_base.cc:1034] running on GCE node
I20240502 14:04:25.157521 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:04:25.157764 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:04:25.157953 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658665157934 us; error 0 us; skew 500 ppm
I20240502 14:04:25.158653 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:04:25.161892 11301 webserver.cc:460] Webserver started at http://127.11.9.66:33739/ using document root <none> and password file <none>
I20240502 14:04:25.162397 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:25.162624 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:25.162973 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:25.164425 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658659912502-11301-0/minicluster-data/ts-1-root/instance:
uuid: "f12a3c49abe14896b2803edbd5662e42"
format_stamp: "Formatted at 2024-05-02 14:04:25 on dist-test-slave-j72s"
I20240502 14:04:25.170233 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20240502 14:04:25.174765 12005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:25.175684 11301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:04:25.175953 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658659912502-11301-0/minicluster-data/ts-1-root
uuid: "f12a3c49abe14896b2803edbd5662e42"
format_stamp: "Formatted at 2024-05-02 14:04:25 on dist-test-slave-j72s"
I20240502 14:04:25.176192 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658659912502-11301-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658659912502-11301-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658659912502-11301-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:25.185122 11994 heartbeater.cc:347] Connected to a master server at 127.11.9.126:36455
I20240502 14:04:25.185457 11994 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:25.186196 11994 heartbeater.cc:511] Master 127.11.9.126:36455 requested a full tablet report, sending...
I20240502 14:04:25.188052 11867 ts_manager.cc:194] Registered new tserver with Master: 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197)
I20240502 14:04:25.190228 11867 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38886
I20240502 14:04:25.192034 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:25.192972 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:25.194358 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:04:25.196401 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:25.196570 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:25.196796 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:25.196935 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:25.233165 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.66:40353
I20240502 14:04:25.233266 12068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.66:40353 every 8 connection(s)
I20240502 14:04:25.237567 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:25.244592 12072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:25.245503 12073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:25.246948 11301 server_base.cc:1034] running on GCE node
W20240502 14:04:25.248208 12075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:25.249187 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:04:25.249399 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:04:25.249610 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658665249592 us; error 0 us; skew 500 ppm
I20240502 14:04:25.249706 12069 heartbeater.cc:347] Connected to a master server at 127.11.9.126:36455
I20240502 14:04:25.250095 12069 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:25.250221 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:04:25.250823 12069 heartbeater.cc:511] Master 127.11.9.126:36455 requested a full tablet report, sending...
I20240502 14:04:25.252668 11867 ts_manager.cc:194] Registered new tserver with Master: f12a3c49abe14896b2803edbd5662e42 (127.11.9.66:40353)
I20240502 14:04:25.252943 11301 webserver.cc:460] Webserver started at http://127.11.9.67:45153/ using document root <none> and password file <none>
I20240502 14:04:25.253419 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:25.253628 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:25.253875 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:25.254207 11867 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38900
I20240502 14:04:25.254915 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658659912502-11301-0/minicluster-data/ts-2-root/instance:
uuid: "80949093e9ab44a8ad5f62462d0d5414"
format_stamp: "Formatted at 2024-05-02 14:04:25 on dist-test-slave-j72s"
I20240502 14:04:25.258852 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20240502 14:04:25.261610 12080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:25.262274 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:04:25.262529 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658659912502-11301-0/minicluster-data/ts-2-root
uuid: "80949093e9ab44a8ad5f62462d0d5414"
format_stamp: "Formatted at 2024-05-02 14:04:25 on dist-test-slave-j72s"
I20240502 14:04:25.262779 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658659912502-11301-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658659912502-11301-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658659912502-11301-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:25.276715 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:25.277660 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:25.279002 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:04:25.281102 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:25.281312 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:25.281560 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:25.281720 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:25.318116 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.67:39977
I20240502 14:04:25.318209 12142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.67:39977 every 8 connection(s)
I20240502 14:04:25.330360 12143 heartbeater.cc:347] Connected to a master server at 127.11.9.126:36455
I20240502 14:04:25.330708 12143 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:25.331383 12143 heartbeater.cc:511] Master 127.11.9.126:36455 requested a full tablet report, sending...
I20240502 14:04:25.333067 11867 ts_manager.cc:194] Registered new tserver with Master: 80949093e9ab44a8ad5f62462d0d5414 (127.11.9.67:39977)
I20240502 14:04:25.333175 11301 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012094244s
I20240502 14:04:25.334365 11867 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38912
W20240502 14:04:26.130280 11898 debug-util.cc:398] Leaking SignalData structure 0x7b08000e3b00 after lost signal to thread 11304
I20240502 14:04:26.192570 11994 heartbeater.cc:503] Master 127.11.9.126:36455 was elected leader, sending a full tablet report...
I20240502 14:04:26.256314 12069 heartbeater.cc:503] Master 127.11.9.126:36455 was elected leader, sending a full tablet report...
I20240502 14:04:26.336450 12143 heartbeater.cc:503] Master 127.11.9.126:36455 was elected leader, sending a full tablet report...
I20240502 14:04:26.364882 11301 test_util.cc:268] Using random seed: 347619263
I20240502 14:04:26.396153 11867 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38924:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:04:26.399711 11867 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:04:26.490804 12108 tablet_service.cc:1467] Processing CreateTablet for tablet fb15de48a4294823b60b2892a4241ad0 (DEFAULT_TABLE table=test-workload [id=c3484e7e7b06438f81cd3d91251b61dc]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 14:04:26.491181 12034 tablet_service.cc:1467] Processing CreateTablet for tablet fb15de48a4294823b60b2892a4241ad0 (DEFAULT_TABLE table=test-workload [id=c3484e7e7b06438f81cd3d91251b61dc]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 14:04:26.492205 12108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb15de48a4294823b60b2892a4241ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:26.492669 12034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb15de48a4294823b60b2892a4241ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:26.495189 12107 tablet_service.cc:1467] Processing CreateTablet for tablet 7284f9efdc5745c1a8f63af14c24d838 (DEFAULT_TABLE table=test-workload [id=c3484e7e7b06438f81cd3d91251b61dc]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 14:04:26.496513 12107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7284f9efdc5745c1a8f63af14c24d838. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:26.496582 12031 tablet_service.cc:1467] Processing CreateTablet for tablet 5509e46f7fc94f32b0f59c7063e87546 (DEFAULT_TABLE table=test-workload [id=c3484e7e7b06438f81cd3d91251b61dc]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 14:04:26.498059 12031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5509e46f7fc94f32b0f59c7063e87546. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:26.501425 12103 tablet_service.cc:1467] Processing CreateTablet for tablet 3c17b90e024548eb89f83fa426c08401 (DEFAULT_TABLE table=test-workload [id=c3484e7e7b06438f81cd3d91251b61dc]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 14:04:26.502732 12103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c17b90e024548eb89f83fa426c08401. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:26.503398 12032 tablet_service.cc:1467] Processing CreateTablet for tablet b0d765fb4e1e49c2b654e9917e89c631 (DEFAULT_TABLE table=test-workload [id=c3484e7e7b06438f81cd3d91251b61dc]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 14:04:26.504237 12105 tablet_service.cc:1467] Processing CreateTablet for tablet 5509e46f7fc94f32b0f59c7063e87546 (DEFAULT_TABLE table=test-workload [id=c3484e7e7b06438f81cd3d91251b61dc]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 14:04:26.504751 12032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d765fb4e1e49c2b654e9917e89c631. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:26.506248 12033 tablet_service.cc:1467] Processing CreateTablet for tablet 7284f9efdc5745c1a8f63af14c24d838 (DEFAULT_TABLE table=test-workload [id=c3484e7e7b06438f81cd3d91251b61dc]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 14:04:26.506678 12104 tablet_service.cc:1467] Processing CreateTablet for tablet 3f82d6f517dc44f18f6d054cd4590124 (DEFAULT_TABLE table=test-workload [id=c3484e7e7b06438f81cd3d91251b61dc]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 14:04:26.508632 12106 tablet_service.cc:1467] Processing CreateTablet for tablet b0d765fb4e1e49c2b654e9917e89c631 (DEFAULT_TABLE table=test-workload [id=c3484e7e7b06438f81cd3d91251b61dc]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 14:04:26.510782 12105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5509e46f7fc94f32b0f59c7063e87546. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:26.517062 12033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7284f9efdc5745c1a8f63af14c24d838. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:26.518085 12104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f82d6f517dc44f18f6d054cd4590124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:26.521852 12106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d765fb4e1e49c2b654e9917e89c631. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:26.522884 12030 tablet_service.cc:1467] Processing CreateTablet for tablet 3f82d6f517dc44f18f6d054cd4590124 (DEFAULT_TABLE table=test-workload [id=c3484e7e7b06438f81cd3d91251b61dc]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 14:04:26.524274 12030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f82d6f517dc44f18f6d054cd4590124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:26.526800 11959 tablet_service.cc:1467] Processing CreateTablet for tablet fb15de48a4294823b60b2892a4241ad0 (DEFAULT_TABLE table=test-workload [id=c3484e7e7b06438f81cd3d91251b61dc]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240502 14:04:26.527227 12029 tablet_service.cc:1467] Processing CreateTablet for tablet 3c17b90e024548eb89f83fa426c08401 (DEFAULT_TABLE table=test-workload [id=c3484e7e7b06438f81cd3d91251b61dc]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 14:04:26.528046 11959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb15de48a4294823b60b2892a4241ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:26.528584 12029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c17b90e024548eb89f83fa426c08401. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:26.531143 11958 tablet_service.cc:1467] Processing CreateTablet for tablet 7284f9efdc5745c1a8f63af14c24d838 (DEFAULT_TABLE table=test-workload [id=c3484e7e7b06438f81cd3d91251b61dc]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240502 14:04:26.532989 11958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7284f9efdc5745c1a8f63af14c24d838. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:26.534668 11956 tablet_service.cc:1467] Processing CreateTablet for tablet 5509e46f7fc94f32b0f59c7063e87546 (DEFAULT_TABLE table=test-workload [id=c3484e7e7b06438f81cd3d91251b61dc]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240502 14:04:26.532920 12102 tablet_service.cc:1467] Processing CreateTablet for tablet 61aee16dda4d415c85536390284665d9 (DEFAULT_TABLE table=test-workload [id=c3484e7e7b06438f81cd3d91251b61dc]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 14:04:26.538470 12101 tablet_service.cc:1467] Processing CreateTablet for tablet 39b0b3c587fa44868a9b1175a2bba670 (DEFAULT_TABLE table=test-workload [id=c3484e7e7b06438f81cd3d91251b61dc]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 14:04:26.540201 12101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39b0b3c587fa44868a9b1175a2bba670. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:26.538234 11953 tablet_service.cc:1467] Processing CreateTablet for tablet 61aee16dda4d415c85536390284665d9 (DEFAULT_TABLE table=test-workload [id=c3484e7e7b06438f81cd3d91251b61dc]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 14:04:26.542924 11957 tablet_service.cc:1467] Processing CreateTablet for tablet b0d765fb4e1e49c2b654e9917e89c631 (DEFAULT_TABLE table=test-workload [id=c3484e7e7b06438f81cd3d91251b61dc]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240502 14:04:26.544797 11957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d765fb4e1e49c2b654e9917e89c631. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:26.547500 11954 tablet_service.cc:1467] Processing CreateTablet for tablet 3c17b90e024548eb89f83fa426c08401 (DEFAULT_TABLE table=test-workload [id=c3484e7e7b06438f81cd3d91251b61dc]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240502 14:04:26.548767 11954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c17b90e024548eb89f83fa426c08401. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:26.531087 12028 tablet_service.cc:1467] Processing CreateTablet for tablet 61aee16dda4d415c85536390284665d9 (DEFAULT_TABLE table=test-workload [id=c3484e7e7b06438f81cd3d91251b61dc]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240502 14:04:26.554392 12028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61aee16dda4d415c85536390284665d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:26.554033 11955 tablet_service.cc:1467] Processing CreateTablet for tablet 3f82d6f517dc44f18f6d054cd4590124 (DEFAULT_TABLE table=test-workload [id=c3484e7e7b06438f81cd3d91251b61dc]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240502 14:04:26.552479 11952 tablet_service.cc:1467] Processing CreateTablet for tablet 39b0b3c587fa44868a9b1175a2bba670 (DEFAULT_TABLE table=test-workload [id=c3484e7e7b06438f81cd3d91251b61dc]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 14:04:26.558472 12102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61aee16dda4d415c85536390284665d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:26.558593 11955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f82d6f517dc44f18f6d054cd4590124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:26.574555 11953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61aee16dda4d415c85536390284665d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:26.576157 11956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5509e46f7fc94f32b0f59c7063e87546. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:26.577683 11952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39b0b3c587fa44868a9b1175a2bba670. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:26.584363 12033 tablet_service.cc:1467] Processing CreateTablet for tablet 39b0b3c587fa44868a9b1175a2bba670 (DEFAULT_TABLE table=test-workload [id=c3484e7e7b06438f81cd3d91251b61dc]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240502 14:04:26.585623 12033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39b0b3c587fa44868a9b1175a2bba670. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:26.591516 12163 tablet_bootstrap.cc:492] T 5509e46f7fc94f32b0f59c7063e87546 P f12a3c49abe14896b2803edbd5662e42: Bootstrap starting.
I20240502 14:04:26.662370 12164 tablet_bootstrap.cc:492] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414: Bootstrap starting.
I20240502 14:04:26.684017 12165 tablet_bootstrap.cc:492] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f: Bootstrap starting.
I20240502 14:04:26.684418 12163 tablet_bootstrap.cc:654] T 5509e46f7fc94f32b0f59c7063e87546 P f12a3c49abe14896b2803edbd5662e42: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:26.689203 12164 tablet_bootstrap.cc:654] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:26.696553 12165 tablet_bootstrap.cc:654] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:26.697021 12163 tablet_bootstrap.cc:492] T 5509e46f7fc94f32b0f59c7063e87546 P f12a3c49abe14896b2803edbd5662e42: No bootstrap required, opened a new log
I20240502 14:04:26.697625 12163 ts_tablet_manager.cc:1367] T 5509e46f7fc94f32b0f59c7063e87546 P f12a3c49abe14896b2803edbd5662e42: Time spent bootstrapping tablet: real 0.106s user 0.017s sys 0.000s
I20240502 14:04:26.699661 12164 tablet_bootstrap.cc:492] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414: No bootstrap required, opened a new log
I20240502 14:04:26.700230 12164 ts_tablet_manager.cc:1367] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.004s
I20240502 14:04:26.700493 12163 raft_consensus.cc:348] T 5509e46f7fc94f32b0f59c7063e87546 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.701776 12163 raft_consensus.cc:374] T 5509e46f7fc94f32b0f59c7063e87546 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:26.702330 12163 raft_consensus.cc:729] T 5509e46f7fc94f32b0f59c7063e87546 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12a3c49abe14896b2803edbd5662e42, State: Initialized, Role: FOLLOWER
I20240502 14:04:26.703033 12164 raft_consensus.cc:348] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.703917 12164 raft_consensus.cc:374] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:26.703418 12163 consensus_queue.cc:260] T 5509e46f7fc94f32b0f59c7063e87546 P f12a3c49abe14896b2803edbd5662e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.704275 12164 raft_consensus.cc:729] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80949093e9ab44a8ad5f62462d0d5414, State: Initialized, Role: FOLLOWER
I20240502 14:04:26.705750 12164 consensus_queue.cc:260] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.707751 12165 tablet_bootstrap.cc:492] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f: No bootstrap required, opened a new log
I20240502 14:04:26.708422 12165 ts_tablet_manager.cc:1367] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.014s
I20240502 14:04:26.711787 12165 raft_consensus.cc:348] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.713009 12165 raft_consensus.cc:374] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:26.713454 12165 raft_consensus.cc:729] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c6d12fd1dd9431e9caac1ca05c3640f, State: Initialized, Role: FOLLOWER
I20240502 14:04:26.714634 12165 consensus_queue.cc:260] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.718273 12163 ts_tablet_manager.cc:1398] T 5509e46f7fc94f32b0f59c7063e87546 P f12a3c49abe14896b2803edbd5662e42: Time spent starting tablet: real 0.020s user 0.003s sys 0.016s
I20240502 14:04:26.719192 12163 tablet_bootstrap.cc:492] T 7284f9efdc5745c1a8f63af14c24d838 P f12a3c49abe14896b2803edbd5662e42: Bootstrap starting.
I20240502 14:04:26.724376 12163 tablet_bootstrap.cc:654] T 7284f9efdc5745c1a8f63af14c24d838 P f12a3c49abe14896b2803edbd5662e42: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:26.735355 12163 tablet_bootstrap.cc:492] T 7284f9efdc5745c1a8f63af14c24d838 P f12a3c49abe14896b2803edbd5662e42: No bootstrap required, opened a new log
I20240502 14:04:26.735908 12163 ts_tablet_manager.cc:1367] T 7284f9efdc5745c1a8f63af14c24d838 P f12a3c49abe14896b2803edbd5662e42: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.009s
I20240502 14:04:26.736711 12164 ts_tablet_manager.cc:1398] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414: Time spent starting tablet: real 0.036s user 0.014s sys 0.019s
I20240502 14:04:26.737774 12164 tablet_bootstrap.cc:492] T 3c17b90e024548eb89f83fa426c08401 P 80949093e9ab44a8ad5f62462d0d5414: Bootstrap starting.
I20240502 14:04:26.738327 12163 raft_consensus.cc:348] T 7284f9efdc5745c1a8f63af14c24d838 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.739200 12163 raft_consensus.cc:374] T 7284f9efdc5745c1a8f63af14c24d838 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:26.739555 12163 raft_consensus.cc:729] T 7284f9efdc5745c1a8f63af14c24d838 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12a3c49abe14896b2803edbd5662e42, State: Initialized, Role: FOLLOWER
I20240502 14:04:26.740408 12163 consensus_queue.cc:260] T 7284f9efdc5745c1a8f63af14c24d838 P f12a3c49abe14896b2803edbd5662e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.742579 12163 ts_tablet_manager.cc:1398] T 7284f9efdc5745c1a8f63af14c24d838 P f12a3c49abe14896b2803edbd5662e42: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 14:04:26.744706 12163 tablet_bootstrap.cc:492] T b0d765fb4e1e49c2b654e9917e89c631 P f12a3c49abe14896b2803edbd5662e42: Bootstrap starting.
I20240502 14:04:26.749972 12163 tablet_bootstrap.cc:654] T b0d765fb4e1e49c2b654e9917e89c631 P f12a3c49abe14896b2803edbd5662e42: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:26.743641 12164 tablet_bootstrap.cc:654] T 3c17b90e024548eb89f83fa426c08401 P 80949093e9ab44a8ad5f62462d0d5414: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:26.754240 12165 ts_tablet_manager.cc:1398] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f: Time spent starting tablet: real 0.045s user 0.008s sys 0.000s
I20240502 14:04:26.757869 12165 tablet_bootstrap.cc:492] T 5509e46f7fc94f32b0f59c7063e87546 P 5c6d12fd1dd9431e9caac1ca05c3640f: Bootstrap starting.
I20240502 14:04:26.763233 12165 tablet_bootstrap.cc:654] T 5509e46f7fc94f32b0f59c7063e87546 P 5c6d12fd1dd9431e9caac1ca05c3640f: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:26.770491 12164 tablet_bootstrap.cc:492] T 3c17b90e024548eb89f83fa426c08401 P 80949093e9ab44a8ad5f62462d0d5414: No bootstrap required, opened a new log
I20240502 14:04:26.770996 12164 ts_tablet_manager.cc:1367] T 3c17b90e024548eb89f83fa426c08401 P 80949093e9ab44a8ad5f62462d0d5414: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.006s
I20240502 14:04:26.773377 12164 raft_consensus.cc:348] T 3c17b90e024548eb89f83fa426c08401 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.774041 12163 tablet_bootstrap.cc:492] T b0d765fb4e1e49c2b654e9917e89c631 P f12a3c49abe14896b2803edbd5662e42: No bootstrap required, opened a new log
I20240502 14:04:26.774209 12164 raft_consensus.cc:374] T 3c17b90e024548eb89f83fa426c08401 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:26.774538 12164 raft_consensus.cc:729] T 3c17b90e024548eb89f83fa426c08401 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80949093e9ab44a8ad5f62462d0d5414, State: Initialized, Role: FOLLOWER
I20240502 14:04:26.774658 12163 ts_tablet_manager.cc:1367] T b0d765fb4e1e49c2b654e9917e89c631 P f12a3c49abe14896b2803edbd5662e42: Time spent bootstrapping tablet: real 0.030s user 0.015s sys 0.011s
I20240502 14:04:26.775424 12164 consensus_queue.cc:260] T 3c17b90e024548eb89f83fa426c08401 P 80949093e9ab44a8ad5f62462d0d5414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.776854 12171 raft_consensus.cc:482] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:26.777786 12164 ts_tablet_manager.cc:1398] T 3c17b90e024548eb89f83fa426c08401 P 80949093e9ab44a8ad5f62462d0d5414: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20240502 14:04:26.777279 12163 raft_consensus.cc:348] T b0d765fb4e1e49c2b654e9917e89c631 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.778227 12163 raft_consensus.cc:374] T b0d765fb4e1e49c2b654e9917e89c631 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:26.778532 12163 raft_consensus.cc:729] T b0d765fb4e1e49c2b654e9917e89c631 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12a3c49abe14896b2803edbd5662e42, State: Initialized, Role: FOLLOWER
I20240502 14:04:26.777309 12171 raft_consensus.cc:504] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.780400 12163 consensus_queue.cc:260] T b0d765fb4e1e49c2b654e9917e89c631 P f12a3c49abe14896b2803edbd5662e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.783526 12171 leader_election.cc:288] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12a3c49abe14896b2803edbd5662e42 (127.11.9.66:40353), 80949093e9ab44a8ad5f62462d0d5414 (127.11.9.67:39977)
I20240502 14:04:26.783602 12163 ts_tablet_manager.cc:1398] T b0d765fb4e1e49c2b654e9917e89c631 P f12a3c49abe14896b2803edbd5662e42: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240502 14:04:26.784731 12164 tablet_bootstrap.cc:492] T 7284f9efdc5745c1a8f63af14c24d838 P 80949093e9ab44a8ad5f62462d0d5414: Bootstrap starting.
I20240502 14:04:26.784997 12163 tablet_bootstrap.cc:492] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42: Bootstrap starting.
I20240502 14:04:26.791386 12163 tablet_bootstrap.cc:654] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:26.782192 12165 tablet_bootstrap.cc:492] T 5509e46f7fc94f32b0f59c7063e87546 P 5c6d12fd1dd9431e9caac1ca05c3640f: No bootstrap required, opened a new log
I20240502 14:04:26.796643 12165 ts_tablet_manager.cc:1367] T 5509e46f7fc94f32b0f59c7063e87546 P 5c6d12fd1dd9431e9caac1ca05c3640f: Time spent bootstrapping tablet: real 0.039s user 0.014s sys 0.003s
I20240502 14:04:26.797091 12164 tablet_bootstrap.cc:654] T 7284f9efdc5745c1a8f63af14c24d838 P 80949093e9ab44a8ad5f62462d0d5414: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:26.807127 12165 raft_consensus.cc:348] T 5509e46f7fc94f32b0f59c7063e87546 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.807974 12165 raft_consensus.cc:374] T 5509e46f7fc94f32b0f59c7063e87546 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:26.808310 12165 raft_consensus.cc:729] T 5509e46f7fc94f32b0f59c7063e87546 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c6d12fd1dd9431e9caac1ca05c3640f, State: Initialized, Role: FOLLOWER
I20240502 14:04:26.808808 12163 tablet_bootstrap.cc:492] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42: No bootstrap required, opened a new log
I20240502 14:04:26.809288 12163 ts_tablet_manager.cc:1367] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42: Time spent bootstrapping tablet: real 0.024s user 0.016s sys 0.000s
I20240502 14:04:26.809137 12165 consensus_queue.cc:260] T 5509e46f7fc94f32b0f59c7063e87546 P 5c6d12fd1dd9431e9caac1ca05c3640f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.812340 12163 raft_consensus.cc:348] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.813154 12163 raft_consensus.cc:374] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:26.813474 12163 raft_consensus.cc:729] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12a3c49abe14896b2803edbd5662e42, State: Initialized, Role: FOLLOWER
I20240502 14:04:26.813707 12164 tablet_bootstrap.cc:492] T 7284f9efdc5745c1a8f63af14c24d838 P 80949093e9ab44a8ad5f62462d0d5414: No bootstrap required, opened a new log
I20240502 14:04:26.814239 12164 ts_tablet_manager.cc:1367] T 7284f9efdc5745c1a8f63af14c24d838 P 80949093e9ab44a8ad5f62462d0d5414: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.009s
I20240502 14:04:26.814388 12163 consensus_queue.cc:260] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.816570 12163 ts_tablet_manager.cc:1398] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 14:04:26.817500 12163 tablet_bootstrap.cc:492] T 3f82d6f517dc44f18f6d054cd4590124 P f12a3c49abe14896b2803edbd5662e42: Bootstrap starting.
I20240502 14:04:26.818140 12165 ts_tablet_manager.cc:1398] T 5509e46f7fc94f32b0f59c7063e87546 P 5c6d12fd1dd9431e9caac1ca05c3640f: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20240502 14:04:26.819041 12044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7284f9efdc5745c1a8f63af14c24d838" candidate_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12a3c49abe14896b2803edbd5662e42" is_pre_election: true
I20240502 14:04:26.822381 12044 raft_consensus.cc:2454] T 7284f9efdc5745c1a8f63af14c24d838 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c6d12fd1dd9431e9caac1ca05c3640f in term 0.
I20240502 14:04:26.823448 12163 tablet_bootstrap.cc:654] T 3f82d6f517dc44f18f6d054cd4590124 P f12a3c49abe14896b2803edbd5662e42: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:26.817227 12164 raft_consensus.cc:348] T 7284f9efdc5745c1a8f63af14c24d838 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.826043 12164 raft_consensus.cc:374] T 7284f9efdc5745c1a8f63af14c24d838 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:26.826449 12164 raft_consensus.cc:729] T 7284f9efdc5745c1a8f63af14c24d838 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80949093e9ab44a8ad5f62462d0d5414, State: Initialized, Role: FOLLOWER
I20240502 14:04:26.827867 12165 tablet_bootstrap.cc:492] T fb15de48a4294823b60b2892a4241ad0 P 5c6d12fd1dd9431e9caac1ca05c3640f: Bootstrap starting.
I20240502 14:04:26.827457 12164 consensus_queue.cc:260] T 7284f9efdc5745c1a8f63af14c24d838 P 80949093e9ab44a8ad5f62462d0d5414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.833585 12165 tablet_bootstrap.cc:654] T fb15de48a4294823b60b2892a4241ad0 P 5c6d12fd1dd9431e9caac1ca05c3640f: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:26.834195 12164 ts_tablet_manager.cc:1398] T 7284f9efdc5745c1a8f63af14c24d838 P 80949093e9ab44a8ad5f62462d0d5414: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20240502 14:04:26.835454 12164 tablet_bootstrap.cc:492] T 61aee16dda4d415c85536390284665d9 P 80949093e9ab44a8ad5f62462d0d5414: Bootstrap starting.
I20240502 14:04:26.838951 12163 tablet_bootstrap.cc:492] T 3f82d6f517dc44f18f6d054cd4590124 P f12a3c49abe14896b2803edbd5662e42: No bootstrap required, opened a new log
I20240502 14:04:26.839531 12163 ts_tablet_manager.cc:1367] T 3f82d6f517dc44f18f6d054cd4590124 P f12a3c49abe14896b2803edbd5662e42: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.008s
I20240502 14:04:26.840680 12164 tablet_bootstrap.cc:654] T 61aee16dda4d415c85536390284665d9 P 80949093e9ab44a8ad5f62462d0d5414: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:26.842159 11933 leader_election.cc:302] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c6d12fd1dd9431e9caac1ca05c3640f, f12a3c49abe14896b2803edbd5662e42; no voters:
I20240502 14:04:26.843057 12171 raft_consensus.cc:2781] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:26.843612 12171 raft_consensus.cc:482] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:26.842161 12163 raft_consensus.cc:348] T 3f82d6f517dc44f18f6d054cd4590124 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.843955 12171 raft_consensus.cc:3037] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:26.844010 12163 raft_consensus.cc:374] T 3f82d6f517dc44f18f6d054cd4590124 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:26.844452 12163 raft_consensus.cc:729] T 3f82d6f517dc44f18f6d054cd4590124 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12a3c49abe14896b2803edbd5662e42, State: Initialized, Role: FOLLOWER
I20240502 14:04:26.845256 12163 consensus_queue.cc:260] T 3f82d6f517dc44f18f6d054cd4590124 P f12a3c49abe14896b2803edbd5662e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.847594 12163 ts_tablet_manager.cc:1398] T 3f82d6f517dc44f18f6d054cd4590124 P f12a3c49abe14896b2803edbd5662e42: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240502 14:04:26.848536 12163 tablet_bootstrap.cc:492] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42: Bootstrap starting.
I20240502 14:04:26.849767 12165 tablet_bootstrap.cc:492] T fb15de48a4294823b60b2892a4241ad0 P 5c6d12fd1dd9431e9caac1ca05c3640f: No bootstrap required, opened a new log
I20240502 14:04:26.851866 12165 ts_tablet_manager.cc:1367] T fb15de48a4294823b60b2892a4241ad0 P 5c6d12fd1dd9431e9caac1ca05c3640f: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.005s
I20240502 14:04:26.850963 12171 raft_consensus.cc:504] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.853829 12163 tablet_bootstrap.cc:654] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:26.855660 12044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7284f9efdc5745c1a8f63af14c24d838" candidate_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12a3c49abe14896b2803edbd5662e42"
I20240502 14:04:26.855842 12165 raft_consensus.cc:348] T fb15de48a4294823b60b2892a4241ad0 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } }
I20240502 14:04:26.856478 12165 raft_consensus.cc:374] T fb15de48a4294823b60b2892a4241ad0 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:26.856207 12044 raft_consensus.cc:3037] T 7284f9efdc5745c1a8f63af14c24d838 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:26.856771 12165 raft_consensus.cc:729] T fb15de48a4294823b60b2892a4241ad0 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c6d12fd1dd9431e9caac1ca05c3640f, State: Initialized, Role: FOLLOWER
I20240502 14:04:26.858125 12171 leader_election.cc:288] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f [CANDIDATE]: Term 1 election: Requested vote from peers f12a3c49abe14896b2803edbd5662e42 (127.11.9.66:40353), 80949093e9ab44a8ad5f62462d0d5414 (127.11.9.67:39977)
I20240502 14:04:26.858727 12165 consensus_queue.cc:260] T fb15de48a4294823b60b2892a4241ad0 P 5c6d12fd1dd9431e9caac1ca05c3640f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } }
I20240502 14:04:26.860787 12165 ts_tablet_manager.cc:1398] T fb15de48a4294823b60b2892a4241ad0 P 5c6d12fd1dd9431e9caac1ca05c3640f: Time spent starting tablet: real 0.009s user 0.003s sys 0.001s
I20240502 14:04:26.865687 12165 tablet_bootstrap.cc:492] T 39b0b3c587fa44868a9b1175a2bba670 P 5c6d12fd1dd9431e9caac1ca05c3640f: Bootstrap starting.
I20240502 14:04:26.865696 12163 tablet_bootstrap.cc:492] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42: No bootstrap required, opened a new log
I20240502 14:04:26.866614 12163 ts_tablet_manager.cc:1367] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.010s
I20240502 14:04:26.868759 12169 raft_consensus.cc:482] T b0d765fb4e1e49c2b654e9917e89c631 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:26.869189 12169 raft_consensus.cc:504] T b0d765fb4e1e49c2b654e9917e89c631 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.869833 12163 raft_consensus.cc:348] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.871063 12163 raft_consensus.cc:374] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:26.871414 12163 raft_consensus.cc:729] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12a3c49abe14896b2803edbd5662e42, State: Initialized, Role: FOLLOWER
I20240502 14:04:26.872226 12163 consensus_queue.cc:260] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.872258 12169 leader_election.cc:288] T b0d765fb4e1e49c2b654e9917e89c631 P f12a3c49abe14896b2803edbd5662e42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197), 80949093e9ab44a8ad5f62462d0d5414 (127.11.9.67:39977)
I20240502 14:04:26.868734 12044 raft_consensus.cc:2454] T 7284f9efdc5745c1a8f63af14c24d838 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c6d12fd1dd9431e9caac1ca05c3640f in term 1.
I20240502 14:04:26.891316 12163 ts_tablet_manager.cc:1398] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42: Time spent starting tablet: real 0.024s user 0.011s sys 0.009s
I20240502 14:04:26.892347 12117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7284f9efdc5745c1a8f63af14c24d838" candidate_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80949093e9ab44a8ad5f62462d0d5414"
I20240502 14:04:26.892933 12163 tablet_bootstrap.cc:492] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42: Bootstrap starting.
I20240502 14:04:26.892800 12118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7284f9efdc5745c1a8f63af14c24d838" candidate_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80949093e9ab44a8ad5f62462d0d5414" is_pre_election: true
I20240502 14:04:26.894340 12117 raft_consensus.cc:3037] T 7284f9efdc5745c1a8f63af14c24d838 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:26.897207 12165 tablet_bootstrap.cc:654] T 39b0b3c587fa44868a9b1175a2bba670 P 5c6d12fd1dd9431e9caac1ca05c3640f: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:26.899565 12163 tablet_bootstrap.cc:654] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:26.912102 11933 leader_election.cc:302] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c6d12fd1dd9431e9caac1ca05c3640f, f12a3c49abe14896b2803edbd5662e42; no voters:
I20240502 14:04:26.913283 12171 raft_consensus.cc:2781] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:26.923193 12170 raft_consensus.cc:482] T 3c17b90e024548eb89f83fa426c08401 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:26.923687 12170 raft_consensus.cc:504] T 3c17b90e024548eb89f83fa426c08401 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.928170 12184 raft_consensus.cc:482] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 14:04:26.930260 11915 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 14:04:26.931392 12165 tablet_bootstrap.cc:492] T 39b0b3c587fa44868a9b1175a2bba670 P 5c6d12fd1dd9431e9caac1ca05c3640f: No bootstrap required, opened a new log
I20240502 14:04:26.931789 12170 leader_election.cc:288] T 3c17b90e024548eb89f83fa426c08401 P 80949093e9ab44a8ad5f62462d0d5414 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197), f12a3c49abe14896b2803edbd5662e42 (127.11.9.66:40353)
I20240502 14:04:26.932039 12118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d765fb4e1e49c2b654e9917e89c631" candidate_uuid: "f12a3c49abe14896b2803edbd5662e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80949093e9ab44a8ad5f62462d0d5414" is_pre_election: true
I20240502 14:04:26.931949 12165 ts_tablet_manager.cc:1367] T 39b0b3c587fa44868a9b1175a2bba670 P 5c6d12fd1dd9431e9caac1ca05c3640f: Time spent bootstrapping tablet: real 0.066s user 0.016s sys 0.016s
I20240502 14:04:26.928822 12184 raft_consensus.cc:504] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.926451 12171 raft_consensus.cc:686] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 LEADER]: Becoming Leader. State: Replica: 5c6d12fd1dd9431e9caac1ca05c3640f, State: Running, Role: LEADER
I20240502 14:04:26.942534 12184 leader_election.cc:288] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197), 80949093e9ab44a8ad5f62462d0d5414 (127.11.9.67:39977)
I20240502 14:04:26.942983 12164 tablet_bootstrap.cc:492] T 61aee16dda4d415c85536390284665d9 P 80949093e9ab44a8ad5f62462d0d5414: No bootstrap required, opened a new log
I20240502 14:04:26.946974 12164 ts_tablet_manager.cc:1367] T 61aee16dda4d415c85536390284665d9 P 80949093e9ab44a8ad5f62462d0d5414: Time spent bootstrapping tablet: real 0.112s user 0.011s sys 0.012s
I20240502 14:04:26.946759 12170 raft_consensus.cc:482] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:26.948799 12171 consensus_queue.cc:237] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.949473 11920 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:04:26.950634 12170 raft_consensus.cc:504] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.954327 12163 tablet_bootstrap.cc:492] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42: No bootstrap required, opened a new log
I20240502 14:04:26.954706 12163 ts_tablet_manager.cc:1367] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42: Time spent bootstrapping tablet: real 0.062s user 0.023s sys 0.013s
I20240502 14:04:26.954841 12170 leader_election.cc:288] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197), f12a3c49abe14896b2803edbd5662e42 (127.11.9.66:40353)
I20240502 14:04:26.955492 12164 raft_consensus.cc:348] T 61aee16dda4d415c85536390284665d9 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.955955 12164 raft_consensus.cc:374] T 61aee16dda4d415c85536390284665d9 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:26.956144 12164 raft_consensus.cc:729] T 61aee16dda4d415c85536390284665d9 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80949093e9ab44a8ad5f62462d0d5414, State: Initialized, Role: FOLLOWER
I20240502 14:04:26.956544 12163 raft_consensus.cc:348] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } }
I20240502 14:04:26.956607 12164 consensus_queue.cc:260] T 61aee16dda4d415c85536390284665d9 P 80949093e9ab44a8ad5f62462d0d5414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.957108 12163 raft_consensus.cc:374] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:26.957307 12163 raft_consensus.cc:729] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12a3c49abe14896b2803edbd5662e42, State: Initialized, Role: FOLLOWER
I20240502 14:04:26.958449 11920 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:04:26.957831 12163 consensus_queue.cc:260] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } }
W20240502 14:04:26.960263 12010 leader_election.cc:341] T b0d765fb4e1e49c2b654e9917e89c631 P f12a3c49abe14896b2803edbd5662e42 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80949093e9ab44a8ad5f62462d0d5414 (127.11.9.67:39977): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:04:26.960521 12163 ts_tablet_manager.cc:1398] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 14:04:26.947461 12117 raft_consensus.cc:2454] T 7284f9efdc5745c1a8f63af14c24d838 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c6d12fd1dd9431e9caac1ca05c3640f in term 1.
I20240502 14:04:26.962774 12163 tablet_bootstrap.cc:492] T 3c17b90e024548eb89f83fa426c08401 P f12a3c49abe14896b2803edbd5662e42: Bootstrap starting.
I20240502 14:04:26.965843 11969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d765fb4e1e49c2b654e9917e89c631" candidate_uuid: "f12a3c49abe14896b2803edbd5662e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" is_pre_election: true
I20240502 14:04:26.966727 12117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61aee16dda4d415c85536390284665d9" candidate_uuid: "f12a3c49abe14896b2803edbd5662e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80949093e9ab44a8ad5f62462d0d5414" is_pre_election: true
I20240502 14:04:26.967558 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61aee16dda4d415c85536390284665d9" candidate_uuid: "f12a3c49abe14896b2803edbd5662e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" is_pre_election: true
I20240502 14:04:26.967636 12117 raft_consensus.cc:2454] T 61aee16dda4d415c85536390284665d9 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12a3c49abe14896b2803edbd5662e42 in term 0.
I20240502 14:04:26.966753 12164 ts_tablet_manager.cc:1398] T 61aee16dda4d415c85536390284665d9 P 80949093e9ab44a8ad5f62462d0d5414: Time spent starting tablet: real 0.017s user 0.004s sys 0.000s
I20240502 14:04:26.969348 12010 leader_election.cc:302] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80949093e9ab44a8ad5f62462d0d5414, f12a3c49abe14896b2803edbd5662e42; no voters:
W20240502 14:04:26.970849 12009 leader_election.cc:341] T b0d765fb4e1e49c2b654e9917e89c631 P f12a3c49abe14896b2803edbd5662e42 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:04:26.971621 12009 leader_election.cc:302] T b0d765fb4e1e49c2b654e9917e89c631 P f12a3c49abe14896b2803edbd5662e42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12a3c49abe14896b2803edbd5662e42; no voters: 5c6d12fd1dd9431e9caac1ca05c3640f, 80949093e9ab44a8ad5f62462d0d5414
I20240502 14:04:26.972497 12184 raft_consensus.cc:2726] T b0d765fb4e1e49c2b654e9917e89c631 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:04:26.972463 12164 tablet_bootstrap.cc:492] T 39b0b3c587fa44868a9b1175a2bba670 P 80949093e9ab44a8ad5f62462d0d5414: Bootstrap starting.
I20240502 14:04:26.974254 12163 tablet_bootstrap.cc:654] T 3c17b90e024548eb89f83fa426c08401 P f12a3c49abe14896b2803edbd5662e42: Neither blocks nor log segments found. Creating new log.
W20240502 14:04:26.975272 12009 leader_election.cc:341] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:04:26.976831 12165 raft_consensus.cc:348] T 39b0b3c587fa44868a9b1175a2bba670 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.978346 12165 raft_consensus.cc:374] T 39b0b3c587fa44868a9b1175a2bba670 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:26.978709 12165 raft_consensus.cc:729] T 39b0b3c587fa44868a9b1175a2bba670 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c6d12fd1dd9431e9caac1ca05c3640f, State: Initialized, Role: FOLLOWER
I20240502 14:04:26.979569 12165 consensus_queue.cc:260] T 39b0b3c587fa44868a9b1175a2bba670 P 5c6d12fd1dd9431e9caac1ca05c3640f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:26.982638 12165 ts_tablet_manager.cc:1398] T 39b0b3c587fa44868a9b1175a2bba670 P 5c6d12fd1dd9431e9caac1ca05c3640f: Time spent starting tablet: real 0.050s user 0.008s sys 0.000s
I20240502 14:04:26.983867 12165 tablet_bootstrap.cc:492] T 61aee16dda4d415c85536390284665d9 P 5c6d12fd1dd9431e9caac1ca05c3640f: Bootstrap starting.
I20240502 14:04:26.987581 12184 raft_consensus.cc:2781] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:26.988081 12184 raft_consensus.cc:482] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:26.988495 12184 raft_consensus.cc:3037] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:27.002687 12169 raft_consensus.cc:482] T 3f82d6f517dc44f18f6d054cd4590124 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:26.998826 11866 catalog_manager.cc:5484] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65). New cstate: current_term: 1 leader_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } health_report { overall_health: UNKNOWN } } }
I20240502 14:04:27.003204 12169 raft_consensus.cc:504] T 3f82d6f517dc44f18f6d054cd4590124 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.005673 12165 tablet_bootstrap.cc:654] T 61aee16dda4d415c85536390284665d9 P 5c6d12fd1dd9431e9caac1ca05c3640f: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:27.007134 12169 leader_election.cc:288] T 3f82d6f517dc44f18f6d054cd4590124 P f12a3c49abe14896b2803edbd5662e42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197), 80949093e9ab44a8ad5f62462d0d5414 (127.11.9.67:39977)
I20240502 14:04:27.007755 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f82d6f517dc44f18f6d054cd4590124" candidate_uuid: "f12a3c49abe14896b2803edbd5662e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" is_pre_election: true
W20240502 14:04:27.010704 12009 leader_election.cc:341] T 3f82d6f517dc44f18f6d054cd4590124 P f12a3c49abe14896b2803edbd5662e42 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:04:27.011996 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c17b90e024548eb89f83fa426c08401" candidate_uuid: "80949093e9ab44a8ad5f62462d0d5414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" is_pre_election: true
I20240502 14:04:27.012398 11969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5509e46f7fc94f32b0f59c7063e87546" candidate_uuid: "80949093e9ab44a8ad5f62462d0d5414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" is_pre_election: true
W20240502 14:04:27.013636 12083 leader_election.cc:341] T 3c17b90e024548eb89f83fa426c08401 P 80949093e9ab44a8ad5f62462d0d5414 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:04:27.013288 11969 raft_consensus.cc:2454] T 5509e46f7fc94f32b0f59c7063e87546 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80949093e9ab44a8ad5f62462d0d5414 in term 0.
I20240502 14:04:27.014515 12117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f82d6f517dc44f18f6d054cd4590124" candidate_uuid: "f12a3c49abe14896b2803edbd5662e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80949093e9ab44a8ad5f62462d0d5414" is_pre_election: true
W20240502 14:04:27.016124 12010 leader_election.cc:341] T 3f82d6f517dc44f18f6d054cd4590124 P f12a3c49abe14896b2803edbd5662e42 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80949093e9ab44a8ad5f62462d0d5414 (127.11.9.67:39977): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:04:27.016036 12083 leader_election.cc:302] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c6d12fd1dd9431e9caac1ca05c3640f, 80949093e9ab44a8ad5f62462d0d5414; no voters:
I20240502 14:04:27.012423 12164 tablet_bootstrap.cc:654] T 39b0b3c587fa44868a9b1175a2bba670 P 80949093e9ab44a8ad5f62462d0d5414: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:27.017153 12170 raft_consensus.cc:2781] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:27.017537 12170 raft_consensus.cc:482] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:27.018087 12170 raft_consensus.cc:3037] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:27.018350 12010 leader_election.cc:302] T 3f82d6f517dc44f18f6d054cd4590124 P f12a3c49abe14896b2803edbd5662e42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f12a3c49abe14896b2803edbd5662e42; no voters: 5c6d12fd1dd9431e9caac1ca05c3640f, 80949093e9ab44a8ad5f62462d0d5414
I20240502 14:04:27.019577 12169 raft_consensus.cc:2726] T 3f82d6f517dc44f18f6d054cd4590124 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:04:27.019691 12184 raft_consensus.cc:504] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.021987 12184 leader_election.cc:288] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42 [CANDIDATE]: Term 1 election: Requested vote from peers 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197), 80949093e9ab44a8ad5f62462d0d5414 (127.11.9.67:39977)
I20240502 14:04:27.023425 12117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61aee16dda4d415c85536390284665d9" candidate_uuid: "f12a3c49abe14896b2803edbd5662e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80949093e9ab44a8ad5f62462d0d5414"
I20240502 14:04:27.024009 12117 raft_consensus.cc:3037] T 61aee16dda4d415c85536390284665d9 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:27.024591 11969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61aee16dda4d415c85536390284665d9" candidate_uuid: "f12a3c49abe14896b2803edbd5662e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f"
I20240502 14:04:27.025386 12044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c17b90e024548eb89f83fa426c08401" candidate_uuid: "80949093e9ab44a8ad5f62462d0d5414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12a3c49abe14896b2803edbd5662e42" is_pre_election: true
W20240502 14:04:27.025944 12009 leader_election.cc:341] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:04:27.027438 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5509e46f7fc94f32b0f59c7063e87546" candidate_uuid: "80949093e9ab44a8ad5f62462d0d5414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12a3c49abe14896b2803edbd5662e42" is_pre_election: true
I20240502 14:04:27.028613 12043 raft_consensus.cc:2454] T 5509e46f7fc94f32b0f59c7063e87546 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80949093e9ab44a8ad5f62462d0d5414 in term 0.
W20240502 14:04:27.029476 12082 leader_election.cc:341] T 3c17b90e024548eb89f83fa426c08401 P 80949093e9ab44a8ad5f62462d0d5414 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f12a3c49abe14896b2803edbd5662e42 (127.11.9.66:40353): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:04:27.029793 12163 tablet_bootstrap.cc:492] T 3c17b90e024548eb89f83fa426c08401 P f12a3c49abe14896b2803edbd5662e42: No bootstrap required, opened a new log
I20240502 14:04:27.029930 12082 leader_election.cc:302] T 3c17b90e024548eb89f83fa426c08401 P 80949093e9ab44a8ad5f62462d0d5414 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80949093e9ab44a8ad5f62462d0d5414; no voters: 5c6d12fd1dd9431e9caac1ca05c3640f, f12a3c49abe14896b2803edbd5662e42
I20240502 14:04:27.030241 12163 ts_tablet_manager.cc:1367] T 3c17b90e024548eb89f83fa426c08401 P f12a3c49abe14896b2803edbd5662e42: Time spent bootstrapping tablet: real 0.068s user 0.014s sys 0.010s
I20240502 14:04:27.031387 12184 raft_consensus.cc:482] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:27.034583 12184 raft_consensus.cc:504] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } }
I20240502 14:04:27.036437 12163 raft_consensus.cc:348] T 3c17b90e024548eb89f83fa426c08401 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.037122 12184 leader_election.cc:288] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80949093e9ab44a8ad5f62462d0d5414 (127.11.9.67:39977), 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197)
I20240502 14:04:27.037351 12163 raft_consensus.cc:374] T 3c17b90e024548eb89f83fa426c08401 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:27.040020 12163 raft_consensus.cc:729] T 3c17b90e024548eb89f83fa426c08401 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12a3c49abe14896b2803edbd5662e42, State: Initialized, Role: FOLLOWER
I20240502 14:04:27.040892 12163 consensus_queue.cc:260] T 3c17b90e024548eb89f83fa426c08401 P f12a3c49abe14896b2803edbd5662e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.042130 11969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb15de48a4294823b60b2892a4241ad0" candidate_uuid: "f12a3c49abe14896b2803edbd5662e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" is_pre_election: true
I20240502 14:04:27.044066 11969 raft_consensus.cc:2454] T fb15de48a4294823b60b2892a4241ad0 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12a3c49abe14896b2803edbd5662e42 in term 0.
I20240502 14:04:27.045627 12009 leader_election.cc:302] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c6d12fd1dd9431e9caac1ca05c3640f, f12a3c49abe14896b2803edbd5662e42; no voters:
I20240502 14:04:27.047173 12184 raft_consensus.cc:2781] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:27.047528 12184 raft_consensus.cc:482] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:27.047997 12163 ts_tablet_manager.cc:1398] T 3c17b90e024548eb89f83fa426c08401 P f12a3c49abe14896b2803edbd5662e42: Time spent starting tablet: real 0.017s user 0.008s sys 0.000s
I20240502 14:04:27.048187 12184 raft_consensus.cc:3037] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:27.050554 12118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb15de48a4294823b60b2892a4241ad0" candidate_uuid: "f12a3c49abe14896b2803edbd5662e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80949093e9ab44a8ad5f62462d0d5414" is_pre_election: true
I20240502 14:04:27.055218 12117 raft_consensus.cc:2454] T 61aee16dda4d415c85536390284665d9 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12a3c49abe14896b2803edbd5662e42 in term 1.
I20240502 14:04:27.056437 12170 raft_consensus.cc:504] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.059535 11969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5509e46f7fc94f32b0f59c7063e87546" candidate_uuid: "80949093e9ab44a8ad5f62462d0d5414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f"
I20240502 14:04:27.059343 12184 raft_consensus.cc:504] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } }
I20240502 14:04:27.060148 11969 raft_consensus.cc:3037] T 5509e46f7fc94f32b0f59c7063e87546 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:27.061654 12184 leader_election.cc:288] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42 [CANDIDATE]: Term 1 election: Requested vote from peers 80949093e9ab44a8ad5f62462d0d5414 (127.11.9.67:39977), 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197)
I20240502 14:04:27.063117 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb15de48a4294823b60b2892a4241ad0" candidate_uuid: "f12a3c49abe14896b2803edbd5662e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f"
I20240502 14:04:27.062934 12170 leader_election.cc:288] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 [CANDIDATE]: Term 1 election: Requested vote from peers 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197), f12a3c49abe14896b2803edbd5662e42 (127.11.9.66:40353)
I20240502 14:04:27.063777 11968 raft_consensus.cc:3037] T fb15de48a4294823b60b2892a4241ad0 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Advancing to term 1
W20240502 14:04:27.067682 12010 leader_election.cc:341] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 80949093e9ab44a8ad5f62462d0d5414 (127.11.9.67:39977): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:04:27.068910 12117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb15de48a4294823b60b2892a4241ad0" candidate_uuid: "f12a3c49abe14896b2803edbd5662e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80949093e9ab44a8ad5f62462d0d5414"
I20240502 14:04:27.068956 12010 leader_election.cc:302] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80949093e9ab44a8ad5f62462d0d5414, f12a3c49abe14896b2803edbd5662e42; no voters: 5c6d12fd1dd9431e9caac1ca05c3640f
I20240502 14:04:27.070411 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5509e46f7fc94f32b0f59c7063e87546" candidate_uuid: "80949093e9ab44a8ad5f62462d0d5414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12a3c49abe14896b2803edbd5662e42"
I20240502 14:04:27.071403 12043 raft_consensus.cc:3037] T 5509e46f7fc94f32b0f59c7063e87546 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:27.073943 12184 raft_consensus.cc:2781] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Leader election won for term 1
W20240502 14:04:27.073753 12010 leader_election.cc:341] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 80949093e9ab44a8ad5f62462d0d5414 (127.11.9.67:39977): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:04:27.076134 12184 raft_consensus.cc:686] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42 [term 1 LEADER]: Becoming Leader. State: Replica: f12a3c49abe14896b2803edbd5662e42, State: Running, Role: LEADER
I20240502 14:04:27.076659 11969 raft_consensus.cc:2454] T 5509e46f7fc94f32b0f59c7063e87546 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80949093e9ab44a8ad5f62462d0d5414 in term 1.
I20240502 14:04:27.077656 12198 raft_consensus.cc:2726] T 3c17b90e024548eb89f83fa426c08401 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:04:27.076936 12184 consensus_queue.cc:237] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.080550 12083 leader_election.cc:302] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c6d12fd1dd9431e9caac1ca05c3640f, 80949093e9ab44a8ad5f62462d0d5414; no voters:
I20240502 14:04:27.081282 12198 raft_consensus.cc:2781] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:27.081801 12198 raft_consensus.cc:686] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 LEADER]: Becoming Leader. State: Replica: 80949093e9ab44a8ad5f62462d0d5414, State: Running, Role: LEADER
I20240502 14:04:27.082576 12198 consensus_queue.cc:237] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.084203 12043 raft_consensus.cc:2454] T 5509e46f7fc94f32b0f59c7063e87546 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80949093e9ab44a8ad5f62462d0d5414 in term 1.
I20240502 14:04:27.086198 12165 tablet_bootstrap.cc:492] T 61aee16dda4d415c85536390284665d9 P 5c6d12fd1dd9431e9caac1ca05c3640f: No bootstrap required, opened a new log
I20240502 14:04:27.086669 12165 ts_tablet_manager.cc:1367] T 61aee16dda4d415c85536390284665d9 P 5c6d12fd1dd9431e9caac1ca05c3640f: Time spent bootstrapping tablet: real 0.103s user 0.006s sys 0.022s
I20240502 14:04:27.080124 11968 raft_consensus.cc:2454] T fb15de48a4294823b60b2892a4241ad0 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12a3c49abe14896b2803edbd5662e42 in term 1.
I20240502 14:04:27.087916 12009 leader_election.cc:302] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c6d12fd1dd9431e9caac1ca05c3640f, f12a3c49abe14896b2803edbd5662e42; no voters: 80949093e9ab44a8ad5f62462d0d5414
I20240502 14:04:27.088971 12169 raft_consensus.cc:2781] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:27.089380 12165 raft_consensus.cc:348] T 61aee16dda4d415c85536390284665d9 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.094502 12165 raft_consensus.cc:374] T 61aee16dda4d415c85536390284665d9 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:27.094955 12165 raft_consensus.cc:729] T 61aee16dda4d415c85536390284665d9 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c6d12fd1dd9431e9caac1ca05c3640f, State: Initialized, Role: FOLLOWER
I20240502 14:04:27.095964 12164 tablet_bootstrap.cc:492] T 39b0b3c587fa44868a9b1175a2bba670 P 80949093e9ab44a8ad5f62462d0d5414: No bootstrap required, opened a new log
I20240502 14:04:27.097402 12164 ts_tablet_manager.cc:1367] T 39b0b3c587fa44868a9b1175a2bba670 P 80949093e9ab44a8ad5f62462d0d5414: Time spent bootstrapping tablet: real 0.125s user 0.013s sys 0.006s
I20240502 14:04:27.096122 12165 consensus_queue.cc:260] T 61aee16dda4d415c85536390284665d9 P 5c6d12fd1dd9431e9caac1ca05c3640f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.097921 12169 raft_consensus.cc:686] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42 [term 1 LEADER]: Becoming Leader. State: Replica: f12a3c49abe14896b2803edbd5662e42, State: Running, Role: LEADER
I20240502 14:04:27.102123 12164 raft_consensus.cc:348] T 39b0b3c587fa44868a9b1175a2bba670 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.103597 12164 raft_consensus.cc:374] T 39b0b3c587fa44868a9b1175a2bba670 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:27.106856 12169 consensus_queue.cc:237] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } }
I20240502 14:04:27.106915 11864 catalog_manager.cc:5484] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42 reported cstate change: term changed from 0 to 1, leader changed from <none> to f12a3c49abe14896b2803edbd5662e42 (127.11.9.66). New cstate: current_term: 1 leader_uuid: "f12a3c49abe14896b2803edbd5662e42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } health_report { overall_health: UNKNOWN } } }
I20240502 14:04:27.108883 12165 ts_tablet_manager.cc:1398] T 61aee16dda4d415c85536390284665d9 P 5c6d12fd1dd9431e9caac1ca05c3640f: Time spent starting tablet: real 0.022s user 0.004s sys 0.003s
I20240502 14:04:27.109197 11867 catalog_manager.cc:5484] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80949093e9ab44a8ad5f62462d0d5414 (127.11.9.67). New cstate: current_term: 1 leader_uuid: "80949093e9ab44a8ad5f62462d0d5414" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } health_report { overall_health: HEALTHY } } }
I20240502 14:04:27.104311 12164 raft_consensus.cc:729] T 39b0b3c587fa44868a9b1175a2bba670 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80949093e9ab44a8ad5f62462d0d5414, State: Initialized, Role: FOLLOWER
I20240502 14:04:27.111588 12165 tablet_bootstrap.cc:492] T b0d765fb4e1e49c2b654e9917e89c631 P 5c6d12fd1dd9431e9caac1ca05c3640f: Bootstrap starting.
I20240502 14:04:27.111425 12164 consensus_queue.cc:260] T 39b0b3c587fa44868a9b1175a2bba670 P 80949093e9ab44a8ad5f62462d0d5414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.116354 12164 ts_tablet_manager.cc:1398] T 39b0b3c587fa44868a9b1175a2bba670 P 80949093e9ab44a8ad5f62462d0d5414: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20240502 14:04:27.117642 12164 tablet_bootstrap.cc:492] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414: Bootstrap starting.
I20240502 14:04:27.135362 12164 tablet_bootstrap.cc:654] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:27.139426 11864 catalog_manager.cc:5484] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42 reported cstate change: term changed from 0 to 1, leader changed from <none> to f12a3c49abe14896b2803edbd5662e42 (127.11.9.66). New cstate: current_term: 1 leader_uuid: "f12a3c49abe14896b2803edbd5662e42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } health_report { overall_health: UNKNOWN } } }
I20240502 14:04:27.146548 12165 tablet_bootstrap.cc:654] T b0d765fb4e1e49c2b654e9917e89c631 P 5c6d12fd1dd9431e9caac1ca05c3640f: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:27.153105 12164 tablet_bootstrap.cc:492] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414: No bootstrap required, opened a new log
I20240502 14:04:27.153559 12164 ts_tablet_manager.cc:1367] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.005s
I20240502 14:04:27.154762 12165 tablet_bootstrap.cc:492] T b0d765fb4e1e49c2b654e9917e89c631 P 5c6d12fd1dd9431e9caac1ca05c3640f: No bootstrap required, opened a new log
I20240502 14:04:27.155237 12165 ts_tablet_manager.cc:1367] T b0d765fb4e1e49c2b654e9917e89c631 P 5c6d12fd1dd9431e9caac1ca05c3640f: Time spent bootstrapping tablet: real 0.044s user 0.010s sys 0.004s
I20240502 14:04:27.155529 12164 raft_consensus.cc:348] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.156018 12164 raft_consensus.cc:374] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:27.156204 12164 raft_consensus.cc:729] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80949093e9ab44a8ad5f62462d0d5414, State: Initialized, Role: FOLLOWER
I20240502 14:04:27.156703 12164 consensus_queue.cc:260] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.158006 12165 raft_consensus.cc:348] T b0d765fb4e1e49c2b654e9917e89c631 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.158612 12164 ts_tablet_manager.cc:1398] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:04:27.158692 12165 raft_consensus.cc:374] T b0d765fb4e1e49c2b654e9917e89c631 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:27.159145 12165 raft_consensus.cc:729] T b0d765fb4e1e49c2b654e9917e89c631 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c6d12fd1dd9431e9caac1ca05c3640f, State: Initialized, Role: FOLLOWER
I20240502 14:04:27.159901 12164 tablet_bootstrap.cc:492] T fb15de48a4294823b60b2892a4241ad0 P 80949093e9ab44a8ad5f62462d0d5414: Bootstrap starting.
I20240502 14:04:27.160015 12165 consensus_queue.cc:260] T b0d765fb4e1e49c2b654e9917e89c631 P 5c6d12fd1dd9431e9caac1ca05c3640f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.160616 12171 raft_consensus.cc:482] T 61aee16dda4d415c85536390284665d9 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:27.161144 12171 raft_consensus.cc:504] T 61aee16dda4d415c85536390284665d9 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.163182 12171 leader_election.cc:288] T 61aee16dda4d415c85536390284665d9 P 5c6d12fd1dd9431e9caac1ca05c3640f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12a3c49abe14896b2803edbd5662e42 (127.11.9.66:40353), 80949093e9ab44a8ad5f62462d0d5414 (127.11.9.67:39977)
I20240502 14:04:27.165028 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61aee16dda4d415c85536390284665d9" candidate_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12a3c49abe14896b2803edbd5662e42" is_pre_election: true
I20240502 14:04:27.165028 12117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61aee16dda4d415c85536390284665d9" candidate_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80949093e9ab44a8ad5f62462d0d5414" is_pre_election: true
I20240502 14:04:27.167027 12164 tablet_bootstrap.cc:654] T fb15de48a4294823b60b2892a4241ad0 P 80949093e9ab44a8ad5f62462d0d5414: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:27.167121 12117 raft_consensus.cc:2379] T 61aee16dda4d415c85536390284665d9 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5c6d12fd1dd9431e9caac1ca05c3640f in current term 1: Already voted for candidate f12a3c49abe14896b2803edbd5662e42 in this term.
I20240502 14:04:27.169284 11933 leader_election.cc:302] T 61aee16dda4d415c85536390284665d9 P 5c6d12fd1dd9431e9caac1ca05c3640f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c6d12fd1dd9431e9caac1ca05c3640f; no voters: 80949093e9ab44a8ad5f62462d0d5414, f12a3c49abe14896b2803edbd5662e42
I20240502 14:04:27.170565 12171 raft_consensus.cc:3037] T 61aee16dda4d415c85536390284665d9 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:27.175634 12165 ts_tablet_manager.cc:1398] T b0d765fb4e1e49c2b654e9917e89c631 P 5c6d12fd1dd9431e9caac1ca05c3640f: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20240502 14:04:27.176926 12165 tablet_bootstrap.cc:492] T 3f82d6f517dc44f18f6d054cd4590124 P 5c6d12fd1dd9431e9caac1ca05c3640f: Bootstrap starting.
I20240502 14:04:27.183928 12165 tablet_bootstrap.cc:654] T 3f82d6f517dc44f18f6d054cd4590124 P 5c6d12fd1dd9431e9caac1ca05c3640f: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:27.187259 12171 raft_consensus.cc:2726] T 61aee16dda4d415c85536390284665d9 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:04:27.189738 12164 tablet_bootstrap.cc:492] T fb15de48a4294823b60b2892a4241ad0 P 80949093e9ab44a8ad5f62462d0d5414: No bootstrap required, opened a new log
I20240502 14:04:27.190230 12164 ts_tablet_manager.cc:1367] T fb15de48a4294823b60b2892a4241ad0 P 80949093e9ab44a8ad5f62462d0d5414: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.016s
I20240502 14:04:27.192067 12165 tablet_bootstrap.cc:492] T 3f82d6f517dc44f18f6d054cd4590124 P 5c6d12fd1dd9431e9caac1ca05c3640f: No bootstrap required, opened a new log
I20240502 14:04:27.192524 12165 ts_tablet_manager.cc:1367] T 3f82d6f517dc44f18f6d054cd4590124 P 5c6d12fd1dd9431e9caac1ca05c3640f: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.003s
I20240502 14:04:27.192689 12164 raft_consensus.cc:348] T fb15de48a4294823b60b2892a4241ad0 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } }
I20240502 14:04:27.193447 12164 raft_consensus.cc:374] T fb15de48a4294823b60b2892a4241ad0 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:27.193816 12164 raft_consensus.cc:729] T fb15de48a4294823b60b2892a4241ad0 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80949093e9ab44a8ad5f62462d0d5414, State: Initialized, Role: FOLLOWER
I20240502 14:04:27.194667 12198 raft_consensus.cc:482] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:27.194504 12164 consensus_queue.cc:260] T fb15de48a4294823b60b2892a4241ad0 P 80949093e9ab44a8ad5f62462d0d5414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } }
I20240502 14:04:27.195144 12198 raft_consensus.cc:504] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.195497 12165 raft_consensus.cc:348] T 3f82d6f517dc44f18f6d054cd4590124 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.196206 12165 raft_consensus.cc:374] T 3f82d6f517dc44f18f6d054cd4590124 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:27.196497 12165 raft_consensus.cc:729] T 3f82d6f517dc44f18f6d054cd4590124 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c6d12fd1dd9431e9caac1ca05c3640f, State: Initialized, Role: FOLLOWER
I20240502 14:04:27.196719 12164 ts_tablet_manager.cc:1398] T fb15de48a4294823b60b2892a4241ad0 P 80949093e9ab44a8ad5f62462d0d5414: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20240502 14:04:27.197221 12198 leader_election.cc:288] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197), f12a3c49abe14896b2803edbd5662e42 (127.11.9.66:40353)
I20240502 14:04:27.197135 12165 consensus_queue.cc:260] T 3f82d6f517dc44f18f6d054cd4590124 P 5c6d12fd1dd9431e9caac1ca05c3640f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.199107 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f82d6f517dc44f18f6d054cd4590124" candidate_uuid: "80949093e9ab44a8ad5f62462d0d5414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" is_pre_election: true
I20240502 14:04:27.199445 12164 tablet_bootstrap.cc:492] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414: Bootstrap starting.
I20240502 14:04:27.199713 12165 ts_tablet_manager.cc:1398] T 3f82d6f517dc44f18f6d054cd4590124 P 5c6d12fd1dd9431e9caac1ca05c3640f: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20240502 14:04:27.199738 11968 raft_consensus.cc:2454] T 3f82d6f517dc44f18f6d054cd4590124 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80949093e9ab44a8ad5f62462d0d5414 in term 0.
I20240502 14:04:27.201748 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f82d6f517dc44f18f6d054cd4590124" candidate_uuid: "80949093e9ab44a8ad5f62462d0d5414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12a3c49abe14896b2803edbd5662e42" is_pre_election: true
I20240502 14:04:27.202375 12043 raft_consensus.cc:2454] T 3f82d6f517dc44f18f6d054cd4590124 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80949093e9ab44a8ad5f62462d0d5414 in term 0.
I20240502 14:04:27.203233 12165 tablet_bootstrap.cc:492] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f: Bootstrap starting.
I20240502 14:04:27.203579 12082 leader_election.cc:302] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80949093e9ab44a8ad5f62462d0d5414, f12a3c49abe14896b2803edbd5662e42; no voters:
I20240502 14:04:27.204414 12170 raft_consensus.cc:2781] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:27.204743 12170 raft_consensus.cc:482] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:27.205065 12170 raft_consensus.cc:3037] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:27.208917 12165 tablet_bootstrap.cc:654] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:27.217871 12170 raft_consensus.cc:504] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.222766 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f82d6f517dc44f18f6d054cd4590124" candidate_uuid: "80949093e9ab44a8ad5f62462d0d5414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f"
I20240502 14:04:27.223408 11968 raft_consensus.cc:3037] T 3f82d6f517dc44f18f6d054cd4590124 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:27.225546 12164 tablet_bootstrap.cc:654] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:27.229983 11968 raft_consensus.cc:2454] T 3f82d6f517dc44f18f6d054cd4590124 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80949093e9ab44a8ad5f62462d0d5414 in term 1.
I20240502 14:04:27.231108 12083 leader_election.cc:302] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c6d12fd1dd9431e9caac1ca05c3640f, 80949093e9ab44a8ad5f62462d0d5414; no voters:
I20240502 14:04:27.232018 12198 raft_consensus.cc:2781] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:27.235150 12170 leader_election.cc:288] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 [CANDIDATE]: Term 1 election: Requested vote from peers 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197), f12a3c49abe14896b2803edbd5662e42 (127.11.9.66:40353)
I20240502 14:04:27.236579 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f82d6f517dc44f18f6d054cd4590124" candidate_uuid: "80949093e9ab44a8ad5f62462d0d5414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12a3c49abe14896b2803edbd5662e42"
I20240502 14:04:27.237344 12043 raft_consensus.cc:3037] T 3f82d6f517dc44f18f6d054cd4590124 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:27.238008 12198 raft_consensus.cc:686] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 LEADER]: Becoming Leader. State: Replica: 80949093e9ab44a8ad5f62462d0d5414, State: Running, Role: LEADER
I20240502 14:04:27.238878 12198 consensus_queue.cc:237] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.245260 12043 raft_consensus.cc:2454] T 3f82d6f517dc44f18f6d054cd4590124 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80949093e9ab44a8ad5f62462d0d5414 in term 1.
I20240502 14:04:27.248018 12165 tablet_bootstrap.cc:492] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f: No bootstrap required, opened a new log
I20240502 14:04:27.248539 12165 ts_tablet_manager.cc:1367] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f: Time spent bootstrapping tablet: real 0.045s user 0.021s sys 0.021s
I20240502 14:04:27.249266 12164 tablet_bootstrap.cc:492] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414: No bootstrap required, opened a new log
I20240502 14:04:27.249760 12164 ts_tablet_manager.cc:1367] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414: Time spent bootstrapping tablet: real 0.050s user 0.010s sys 0.008s
I20240502 14:04:27.250982 12165 raft_consensus.cc:348] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.251660 12165 raft_consensus.cc:374] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:27.251937 12165 raft_consensus.cc:729] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c6d12fd1dd9431e9caac1ca05c3640f, State: Initialized, Role: FOLLOWER
I20240502 14:04:27.252561 12164 raft_consensus.cc:348] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.252727 12165 consensus_queue.cc:260] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.253325 12164 raft_consensus.cc:374] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:27.253708 12164 raft_consensus.cc:729] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80949093e9ab44a8ad5f62462d0d5414, State: Initialized, Role: FOLLOWER
I20240502 14:04:27.254457 12164 consensus_queue.cc:260] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.257161 12165 ts_tablet_manager.cc:1398] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240502 14:04:27.259109 12164 ts_tablet_manager.cc:1398] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240502 14:04:27.261960 11866 catalog_manager.cc:5484] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80949093e9ab44a8ad5f62462d0d5414 (127.11.9.67). New cstate: current_term: 1 leader_uuid: "80949093e9ab44a8ad5f62462d0d5414" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } health_report { overall_health: HEALTHY } } }
I20240502 14:04:27.272979 12169 raft_consensus.cc:482] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:27.273506 12169 raft_consensus.cc:504] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.275574 12169 leader_election.cc:288] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197), 80949093e9ab44a8ad5f62462d0d5414 (127.11.9.67:39977)
I20240502 14:04:27.277881 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b0b3c587fa44868a9b1175a2bba670" candidate_uuid: "f12a3c49abe14896b2803edbd5662e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" is_pre_election: true
I20240502 14:04:27.277814 12117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b0b3c587fa44868a9b1175a2bba670" candidate_uuid: "f12a3c49abe14896b2803edbd5662e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80949093e9ab44a8ad5f62462d0d5414" is_pre_election: true
I20240502 14:04:27.278568 11968 raft_consensus.cc:2454] T 39b0b3c587fa44868a9b1175a2bba670 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12a3c49abe14896b2803edbd5662e42 in term 0.
I20240502 14:04:27.278714 12117 raft_consensus.cc:2454] T 39b0b3c587fa44868a9b1175a2bba670 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f12a3c49abe14896b2803edbd5662e42 in term 0.
I20240502 14:04:27.279732 12010 leader_election.cc:302] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80949093e9ab44a8ad5f62462d0d5414, f12a3c49abe14896b2803edbd5662e42; no voters:
I20240502 14:04:27.280624 12169 raft_consensus.cc:2781] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:27.281021 12169 raft_consensus.cc:482] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:27.281353 12169 raft_consensus.cc:3037] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:27.288120 12169 raft_consensus.cc:504] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.290587 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b0b3c587fa44868a9b1175a2bba670" candidate_uuid: "f12a3c49abe14896b2803edbd5662e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f"
I20240502 14:04:27.291079 11968 raft_consensus.cc:3037] T 39b0b3c587fa44868a9b1175a2bba670 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:27.290946 12117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b0b3c587fa44868a9b1175a2bba670" candidate_uuid: "f12a3c49abe14896b2803edbd5662e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80949093e9ab44a8ad5f62462d0d5414"
I20240502 14:04:27.290366 12169 leader_election.cc:288] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42 [CANDIDATE]: Term 1 election: Requested vote from peers 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197), 80949093e9ab44a8ad5f62462d0d5414 (127.11.9.67:39977)
I20240502 14:04:27.291589 12117 raft_consensus.cc:3037] T 39b0b3c587fa44868a9b1175a2bba670 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:27.295181 11968 raft_consensus.cc:2454] T 39b0b3c587fa44868a9b1175a2bba670 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12a3c49abe14896b2803edbd5662e42 in term 1.
I20240502 14:04:27.295985 12009 leader_election.cc:302] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c6d12fd1dd9431e9caac1ca05c3640f, f12a3c49abe14896b2803edbd5662e42; no voters:
I20240502 14:04:27.296546 12169 raft_consensus.cc:2781] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:27.296887 12169 raft_consensus.cc:686] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42 [term 1 LEADER]: Becoming Leader. State: Replica: f12a3c49abe14896b2803edbd5662e42, State: Running, Role: LEADER
I20240502 14:04:27.297653 12169 consensus_queue.cc:237] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.298385 12117 raft_consensus.cc:2454] T 39b0b3c587fa44868a9b1175a2bba670 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f12a3c49abe14896b2803edbd5662e42 in term 1.
I20240502 14:04:27.304677 11866 catalog_manager.cc:5484] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42 reported cstate change: term changed from 0 to 1, leader changed from <none> to f12a3c49abe14896b2803edbd5662e42 (127.11.9.66). New cstate: current_term: 1 leader_uuid: "f12a3c49abe14896b2803edbd5662e42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } health_report { overall_health: UNKNOWN } } }
I20240502 14:04:27.409719 12198 raft_consensus.cc:482] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:27.410312 12198 raft_consensus.cc:504] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.412941 12198 leader_election.cc:288] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197), f12a3c49abe14896b2803edbd5662e42 (127.11.9.66:40353)
I20240502 14:04:27.413278 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d765fb4e1e49c2b654e9917e89c631" candidate_uuid: "80949093e9ab44a8ad5f62462d0d5414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" is_pre_election: true
I20240502 14:04:27.413944 11968 raft_consensus.cc:2454] T b0d765fb4e1e49c2b654e9917e89c631 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80949093e9ab44a8ad5f62462d0d5414 in term 0.
I20240502 14:04:27.413976 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d765fb4e1e49c2b654e9917e89c631" candidate_uuid: "80949093e9ab44a8ad5f62462d0d5414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12a3c49abe14896b2803edbd5662e42" is_pre_election: true
I20240502 14:04:27.414795 12043 raft_consensus.cc:2454] T b0d765fb4e1e49c2b654e9917e89c631 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80949093e9ab44a8ad5f62462d0d5414 in term 0.
I20240502 14:04:27.414970 12083 leader_election.cc:302] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c6d12fd1dd9431e9caac1ca05c3640f, 80949093e9ab44a8ad5f62462d0d5414; no voters:
I20240502 14:04:27.415694 12198 raft_consensus.cc:2781] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:27.416047 12198 raft_consensus.cc:482] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:27.416364 12198 raft_consensus.cc:3037] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:27.418275 12171 raft_consensus.cc:482] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:27.418620 12171 raft_consensus.cc:504] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.420059 12171 leader_election.cc:288] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12a3c49abe14896b2803edbd5662e42 (127.11.9.66:40353), 80949093e9ab44a8ad5f62462d0d5414 (127.11.9.67:39977)
I20240502 14:04:27.420789 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c17b90e024548eb89f83fa426c08401" candidate_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12a3c49abe14896b2803edbd5662e42" is_pre_election: true
I20240502 14:04:27.421367 12043 raft_consensus.cc:2454] T 3c17b90e024548eb89f83fa426c08401 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c6d12fd1dd9431e9caac1ca05c3640f in term 0.
I20240502 14:04:27.421422 12117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c17b90e024548eb89f83fa426c08401" candidate_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80949093e9ab44a8ad5f62462d0d5414" is_pre_election: true
I20240502 14:04:27.422113 12117 raft_consensus.cc:2454] T 3c17b90e024548eb89f83fa426c08401 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c6d12fd1dd9431e9caac1ca05c3640f in term 0.
I20240502 14:04:27.422173 11933 leader_election.cc:302] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c6d12fd1dd9431e9caac1ca05c3640f, f12a3c49abe14896b2803edbd5662e42; no voters:
I20240502 14:04:27.422892 12171 raft_consensus.cc:2781] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:27.423161 12171 raft_consensus.cc:482] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:27.423503 12171 raft_consensus.cc:3037] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:27.425733 12198 raft_consensus.cc:504] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.427704 12198 leader_election.cc:288] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 [CANDIDATE]: Term 1 election: Requested vote from peers 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197), f12a3c49abe14896b2803edbd5662e42 (127.11.9.66:40353)
I20240502 14:04:27.427488 12171 raft_consensus.cc:504] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.429041 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d765fb4e1e49c2b654e9917e89c631" candidate_uuid: "80949093e9ab44a8ad5f62462d0d5414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12a3c49abe14896b2803edbd5662e42"
I20240502 14:04:27.429448 12171 leader_election.cc:288] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f [CANDIDATE]: Term 1 election: Requested vote from peers f12a3c49abe14896b2803edbd5662e42 (127.11.9.66:40353), 80949093e9ab44a8ad5f62462d0d5414 (127.11.9.67:39977)
I20240502 14:04:27.429708 12043 raft_consensus.cc:3037] T b0d765fb4e1e49c2b654e9917e89c631 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:27.430114 12044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c17b90e024548eb89f83fa426c08401" candidate_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12a3c49abe14896b2803edbd5662e42"
I20240502 14:04:27.430547 12044 raft_consensus.cc:3037] T 3c17b90e024548eb89f83fa426c08401 P f12a3c49abe14896b2803edbd5662e42 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:27.434163 12117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c17b90e024548eb89f83fa426c08401" candidate_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80949093e9ab44a8ad5f62462d0d5414"
I20240502 14:04:27.434638 12117 raft_consensus.cc:3037] T 3c17b90e024548eb89f83fa426c08401 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:27.436693 12043 raft_consensus.cc:2454] T b0d765fb4e1e49c2b654e9917e89c631 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80949093e9ab44a8ad5f62462d0d5414 in term 1.
I20240502 14:04:27.438057 12082 leader_election.cc:302] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80949093e9ab44a8ad5f62462d0d5414, f12a3c49abe14896b2803edbd5662e42; no voters:
I20240502 14:04:27.439361 12198 raft_consensus.cc:2781] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:27.439934 12044 raft_consensus.cc:2454] T 3c17b90e024548eb89f83fa426c08401 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c6d12fd1dd9431e9caac1ca05c3640f in term 1.
I20240502 14:04:27.440608 12198 raft_consensus.cc:686] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 LEADER]: Becoming Leader. State: Replica: 80949093e9ab44a8ad5f62462d0d5414, State: Running, Role: LEADER
I20240502 14:04:27.441092 11933 leader_election.cc:302] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c6d12fd1dd9431e9caac1ca05c3640f, f12a3c49abe14896b2803edbd5662e42; no voters:
I20240502 14:04:27.440227 12117 raft_consensus.cc:2454] T 3c17b90e024548eb89f83fa426c08401 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c6d12fd1dd9431e9caac1ca05c3640f in term 1.
I20240502 14:04:27.440742 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d765fb4e1e49c2b654e9917e89c631" candidate_uuid: "80949093e9ab44a8ad5f62462d0d5414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f"
I20240502 14:04:27.441481 12198 consensus_queue.cc:237] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.442334 11968 raft_consensus.cc:3037] T b0d765fb4e1e49c2b654e9917e89c631 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:27.442775 12171 raft_consensus.cc:2781] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:27.443462 12171 raft_consensus.cc:686] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 LEADER]: Becoming Leader. State: Replica: 5c6d12fd1dd9431e9caac1ca05c3640f, State: Running, Role: LEADER
I20240502 14:04:27.444221 12171 consensus_queue.cc:237] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } }
I20240502 14:04:27.449000 11968 raft_consensus.cc:2454] T b0d765fb4e1e49c2b654e9917e89c631 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80949093e9ab44a8ad5f62462d0d5414 in term 1.
I20240502 14:04:27.455715 11866 catalog_manager.cc:5484] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65). New cstate: current_term: 1 leader_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } health_report { overall_health: UNKNOWN } } }
I20240502 14:04:27.457851 11867 catalog_manager.cc:5484] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80949093e9ab44a8ad5f62462d0d5414 (127.11.9.67). New cstate: current_term: 1 leader_uuid: "80949093e9ab44a8ad5f62462d0d5414" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } health_report { overall_health: HEALTHY } } }
I20240502 14:04:27.468766 12198 raft_consensus.cc:482] T fb15de48a4294823b60b2892a4241ad0 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:27.469194 12198 raft_consensus.cc:504] T fb15de48a4294823b60b2892a4241ad0 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } }
I20240502 14:04:27.470732 12198 leader_election.cc:288] T fb15de48a4294823b60b2892a4241ad0 P 80949093e9ab44a8ad5f62462d0d5414 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12a3c49abe14896b2803edbd5662e42 (127.11.9.66:40353), 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197)
I20240502 14:04:27.471459 12044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb15de48a4294823b60b2892a4241ad0" candidate_uuid: "80949093e9ab44a8ad5f62462d0d5414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12a3c49abe14896b2803edbd5662e42" is_pre_election: true
I20240502 14:04:27.471663 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb15de48a4294823b60b2892a4241ad0" candidate_uuid: "80949093e9ab44a8ad5f62462d0d5414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" is_pre_election: true
I20240502 14:04:27.472379 11968 raft_consensus.cc:2379] T fb15de48a4294823b60b2892a4241ad0 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 80949093e9ab44a8ad5f62462d0d5414 in current term 1: Already voted for candidate f12a3c49abe14896b2803edbd5662e42 in this term.
I20240502 14:04:27.473166 12083 leader_election.cc:302] T fb15de48a4294823b60b2892a4241ad0 P 80949093e9ab44a8ad5f62462d0d5414 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80949093e9ab44a8ad5f62462d0d5414; no voters: 5c6d12fd1dd9431e9caac1ca05c3640f, f12a3c49abe14896b2803edbd5662e42
I20240502 14:04:27.473783 12198 raft_consensus.cc:3037] T fb15de48a4294823b60b2892a4241ad0 P 80949093e9ab44a8ad5f62462d0d5414 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:27.480765 12198 raft_consensus.cc:2726] T fb15de48a4294823b60b2892a4241ad0 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:04:27.528735 12171 consensus_queue.cc:1035] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:27.540860 12171 consensus_queue.cc:1035] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f [LEADER]: Connected to new peer: Peer: permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:27.564353 12169 consensus_queue.cc:1035] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:27.574038 12202 consensus_queue.cc:1035] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:27.608985 12202 consensus_queue.cc:1035] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:27.618861 12169 consensus_queue.cc:1035] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:27.642949 12198 consensus_queue.cc:1035] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:27.648448 12198 consensus_queue.cc:1035] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:27.668879 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:27.696362 12229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:27.698925 12230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:27.708397 11301 server_base.cc:1034] running on GCE node
W20240502 14:04:27.709209 12232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:27.710620 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:04:27.711055 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:04:27.711345 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658667711325 us; error 0 us; skew 500 ppm
I20240502 14:04:27.712203 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:04:27.712251 12198 consensus_queue.cc:1035] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:27.712574 12169 consensus_queue.cc:1035] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:27.738191 11301 webserver.cc:460] Webserver started at http://127.11.9.68:39389/ using document root <none> and password file <none>
I20240502 14:04:27.738967 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:27.739269 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:27.739686 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:27.741214 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658659912502-11301-0/minicluster-data/ts-3-root/instance:
uuid: "a120eeb181c24820a8d17de6f1efd322"
format_stamp: "Formatted at 2024-05-02 14:04:27 on dist-test-slave-j72s"
I20240502 14:04:27.746443 12213 consensus_queue.cc:1035] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:27.755033 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.009s sys 0.003s
I20240502 14:04:27.752975 12169 consensus_queue.cc:1035] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240502 14:04:27.760733 12239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:27.763048 11301 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20240502 14:04:27.763653 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658659912502-11301-0/minicluster-data/ts-3-root
uuid: "a120eeb181c24820a8d17de6f1efd322"
format_stamp: "Formatted at 2024-05-02 14:04:27 on dist-test-slave-j72s"
I20240502 14:04:27.764197 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658659912502-11301-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658659912502-11301-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714658659912502-11301-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:27.822266 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:27.823305 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:27.824906 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:04:27.827281 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:27.827461 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:27.827697 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:27.827836 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:27.854632 12171 consensus_queue.cc:1035] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:04:27.877334 12171 consensus_queue.cc:1035] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f [LEADER]: Connected to new peer: Peer: permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:27.905714 12213 consensus_queue.cc:1035] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:27.924238 12213 consensus_queue.cc:1035] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:04:27.994208 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.68:39447
I20240502 14:04:27.994647 12308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.68:39447 every 8 connection(s)
I20240502 14:04:28.009837 12309 heartbeater.cc:347] Connected to a master server at 127.11.9.126:36455
I20240502 14:04:28.010200 12309 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:28.011118 12309 heartbeater.cc:511] Master 127.11.9.126:36455 requested a full tablet report, sending...
I20240502 14:04:28.012948 11866 ts_manager.cc:194] Registered new tserver with Master: a120eeb181c24820a8d17de6f1efd322 (127.11.9.68:39447)
I20240502 14:04:28.014605 11866 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38926
I20240502 14:04:28.959403 11920 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:04:28.965353 11920 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:04:28.965883 11920 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:04:29.017369 12309 heartbeater.cc:503] Master 127.11.9.126:36455 was elected leader, sending a full tablet report...
I20240502 14:04:30.967083 11920 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:04:30.972595 11920 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:04:30.972990 11920 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:04:32.973793 11920 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:04:32.979223 11920 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:04:32.979663 11920 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:04:33.968077 12117 consensus_queue.cc:237] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } attrs { replace: true } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } }
I20240502 14:04:33.975040 12043 raft_consensus.cc:1261] T 5509e46f7fc94f32b0f59c7063e87546 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Refusing update from remote peer 80949093e9ab44a8ad5f62462d0d5414: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:04:33.975700 11968 raft_consensus.cc:1261] T 5509e46f7fc94f32b0f59c7063e87546 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Refusing update from remote peer 80949093e9ab44a8ad5f62462d0d5414: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:04:33.977490 12170 consensus_queue.cc:1035] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 14:04:33.978693 12317 consensus_queue.cc:1035] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:04:33.997002 12170 raft_consensus.cc:2932] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a120eeb181c24820a8d17de6f1efd322 (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } attrs { replace: true } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } } }
W20240502 14:04:33.998804 12081 consensus_peers.cc:485] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 -> Peer a120eeb181c24820a8d17de6f1efd322 (127.11.9.68:39447): Couldn't send request to peer a120eeb181c24820a8d17de6f1efd322. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5509e46f7fc94f32b0f59c7063e87546. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:04:33.999848 12044 raft_consensus.cc:2932] T 5509e46f7fc94f32b0f59c7063e87546 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a120eeb181c24820a8d17de6f1efd322 (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } attrs { replace: true } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } } }
I20240502 14:04:33.999735 11968 raft_consensus.cc:2932] T 5509e46f7fc94f32b0f59c7063e87546 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a120eeb181c24820a8d17de6f1efd322 (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } attrs { replace: true } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } } }
I20240502 14:04:34.011871 12117 consensus_queue.cc:237] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } attrs { replace: true } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } }
I20240502 14:04:34.019150 12044 raft_consensus.cc:1261] T b0d765fb4e1e49c2b654e9917e89c631 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Refusing update from remote peer 80949093e9ab44a8ad5f62462d0d5414: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:04:34.019284 12081 consensus_peers.cc:485] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 -> Peer a120eeb181c24820a8d17de6f1efd322 (127.11.9.68:39447): Couldn't send request to peer a120eeb181c24820a8d17de6f1efd322. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0d765fb4e1e49c2b654e9917e89c631. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:04:34.020998 12326 consensus_queue.cc:1035] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 14:04:34.019170 11866 catalog_manager.cc:5484] T 5509e46f7fc94f32b0f59c7063e87546 P f12a3c49abe14896b2803edbd5662e42 reported cstate change: config changed from index -1 to 2, NON_VOTER a120eeb181c24820a8d17de6f1efd322 (127.11.9.68) added. New cstate: current_term: 1 leader_uuid: "80949093e9ab44a8ad5f62462d0d5414" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } attrs { replace: true } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } } }
I20240502 14:04:34.024955 11969 raft_consensus.cc:1261] T b0d765fb4e1e49c2b654e9917e89c631 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Refusing update from remote peer 80949093e9ab44a8ad5f62462d0d5414: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:04:34.026962 12326 consensus_queue.cc:1035] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 14:04:34.035141 12317 raft_consensus.cc:2932] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a120eeb181c24820a8d17de6f1efd322 (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } attrs { replace: true } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } } }
I20240502 14:04:34.039304 12044 raft_consensus.cc:2932] T b0d765fb4e1e49c2b654e9917e89c631 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a120eeb181c24820a8d17de6f1efd322 (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } attrs { replace: true } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } } }
I20240502 14:04:34.041231 11969 raft_consensus.cc:2932] T b0d765fb4e1e49c2b654e9917e89c631 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a120eeb181c24820a8d17de6f1efd322 (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } attrs { replace: true } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } } }
I20240502 14:04:34.088513 11969 consensus_queue.cc:237] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } attrs { replace: true } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } }
I20240502 14:04:34.089975 11864 catalog_manager.cc:5484] T b0d765fb4e1e49c2b654e9917e89c631 P f12a3c49abe14896b2803edbd5662e42 reported cstate change: config changed from index -1 to 2, NON_VOTER a120eeb181c24820a8d17de6f1efd322 (127.11.9.68) added. New cstate: current_term: 1 leader_uuid: "80949093e9ab44a8ad5f62462d0d5414" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } attrs { replace: true } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } } }
I20240502 14:04:34.096271 12117 raft_consensus.cc:1261] T 7284f9efdc5745c1a8f63af14c24d838 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Refusing update from remote peer 5c6d12fd1dd9431e9caac1ca05c3640f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:04:34.099277 12315 consensus_queue.cc:1035] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f [LEADER]: Connected to new peer: Peer: permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 14:04:34.115325 12044 raft_consensus.cc:1261] T 7284f9efdc5745c1a8f63af14c24d838 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Refusing update from remote peer 5c6d12fd1dd9431e9caac1ca05c3640f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:04:34.114995 12315 raft_consensus.cc:2932] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a120eeb181c24820a8d17de6f1efd322 (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } attrs { replace: true } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } } }
I20240502 14:04:34.116163 12117 raft_consensus.cc:2932] T 7284f9efdc5745c1a8f63af14c24d838 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a120eeb181c24820a8d17de6f1efd322 (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } attrs { replace: true } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } } }
I20240502 14:04:34.123489 12337 consensus_queue.cc:1035] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240502 14:04:34.125198 11932 consensus_peers.cc:485] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f -> Peer a120eeb181c24820a8d17de6f1efd322 (127.11.9.68:39447): Couldn't send request to peer a120eeb181c24820a8d17de6f1efd322. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7284f9efdc5745c1a8f63af14c24d838. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:04:34.129796 12044 raft_consensus.cc:2932] T 7284f9efdc5745c1a8f63af14c24d838 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a120eeb181c24820a8d17de6f1efd322 (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } attrs { replace: true } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } } }
I20240502 14:04:34.130167 11867 catalog_manager.cc:5484] T 7284f9efdc5745c1a8f63af14c24d838 P 80949093e9ab44a8ad5f62462d0d5414 reported cstate change: config changed from index -1 to 2, NON_VOTER a120eeb181c24820a8d17de6f1efd322 (127.11.9.68) added. New cstate: current_term: 1 leader_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } attrs { replace: true } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } } }
I20240502 14:04:34.142916 12117 consensus_queue.cc:237] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } attrs { replace: true } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } }
W20240502 14:04:34.150283 12081 consensus_peers.cc:485] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 -> Peer a120eeb181c24820a8d17de6f1efd322 (127.11.9.68:39447): Couldn't send request to peer a120eeb181c24820a8d17de6f1efd322. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f82d6f517dc44f18f6d054cd4590124. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:04:34.150508 11969 raft_consensus.cc:1261] T 3f82d6f517dc44f18f6d054cd4590124 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Refusing update from remote peer 80949093e9ab44a8ad5f62462d0d5414: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:04:34.150929 12044 raft_consensus.cc:1261] T 3f82d6f517dc44f18f6d054cd4590124 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Refusing update from remote peer 80949093e9ab44a8ad5f62462d0d5414: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:04:34.151659 12326 consensus_queue.cc:1035] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:04:34.152593 12325 consensus_queue.cc:1035] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:04:34.159476 12326 raft_consensus.cc:2932] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a120eeb181c24820a8d17de6f1efd322 (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } attrs { replace: true } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } } }
I20240502 14:04:34.161181 12044 raft_consensus.cc:2932] T 3f82d6f517dc44f18f6d054cd4590124 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a120eeb181c24820a8d17de6f1efd322 (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } attrs { replace: true } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } } }
I20240502 14:04:34.162978 11969 raft_consensus.cc:2932] T 3f82d6f517dc44f18f6d054cd4590124 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a120eeb181c24820a8d17de6f1efd322 (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } attrs { replace: true } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } } }
I20240502 14:04:34.175323 11866 catalog_manager.cc:5484] T 3f82d6f517dc44f18f6d054cd4590124 P 5c6d12fd1dd9431e9caac1ca05c3640f reported cstate change: config changed from index -1 to 2, NON_VOTER a120eeb181c24820a8d17de6f1efd322 (127.11.9.68) added. New cstate: current_term: 1 leader_uuid: "80949093e9ab44a8ad5f62462d0d5414" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } attrs { replace: true } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } } }
I20240502 14:04:34.396536 12346 ts_tablet_manager.cc:902] T b0d765fb4e1e49c2b654e9917e89c631 P a120eeb181c24820a8d17de6f1efd322: Initiating tablet copy from peer 80949093e9ab44a8ad5f62462d0d5414 (127.11.9.67:39977)
I20240502 14:04:34.400981 12346 tablet_copy_client.cc:308] T b0d765fb4e1e49c2b654e9917e89c631 P a120eeb181c24820a8d17de6f1efd322: tablet copy: Beginning tablet copy session from remote peer at address 127.11.9.67:39977
I20240502 14:04:34.418327 12128 tablet_copy_service.cc:140] P 80949093e9ab44a8ad5f62462d0d5414: Received BeginTabletCopySession request for tablet b0d765fb4e1e49c2b654e9917e89c631 from peer a120eeb181c24820a8d17de6f1efd322 ({username='slave'} at 127.0.0.1:43968)
I20240502 14:04:34.418903 12128 tablet_copy_service.cc:161] P 80949093e9ab44a8ad5f62462d0d5414: Beginning new tablet copy session on tablet b0d765fb4e1e49c2b654e9917e89c631 from peer a120eeb181c24820a8d17de6f1efd322 at {username='slave'} at 127.0.0.1:43968: session id = a120eeb181c24820a8d17de6f1efd322-b0d765fb4e1e49c2b654e9917e89c631
I20240502 14:04:34.426385 12128 tablet_copy_source_session.cc:205] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:04:34.429955 12346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d765fb4e1e49c2b654e9917e89c631. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:34.441391 12346 tablet_copy_client.cc:783] T b0d765fb4e1e49c2b654e9917e89c631 P a120eeb181c24820a8d17de6f1efd322: tablet copy: Starting download of 0 data blocks...
I20240502 14:04:34.442023 12346 tablet_copy_client.cc:647] T b0d765fb4e1e49c2b654e9917e89c631 P a120eeb181c24820a8d17de6f1efd322: tablet copy: Starting download of 1 WAL segments...
I20240502 14:04:34.445946 12346 tablet_copy_client.cc:523] T b0d765fb4e1e49c2b654e9917e89c631 P a120eeb181c24820a8d17de6f1efd322: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:04:34.453774 12346 tablet_bootstrap.cc:492] T b0d765fb4e1e49c2b654e9917e89c631 P a120eeb181c24820a8d17de6f1efd322: Bootstrap starting.
I20240502 14:04:34.473819 12346 tablet_bootstrap.cc:492] T b0d765fb4e1e49c2b654e9917e89c631 P a120eeb181c24820a8d17de6f1efd322: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:04:34.474682 12346 tablet_bootstrap.cc:492] T b0d765fb4e1e49c2b654e9917e89c631 P a120eeb181c24820a8d17de6f1efd322: Bootstrap complete.
I20240502 14:04:34.475325 12346 ts_tablet_manager.cc:1367] T b0d765fb4e1e49c2b654e9917e89c631 P a120eeb181c24820a8d17de6f1efd322: Time spent bootstrapping tablet: real 0.022s user 0.023s sys 0.001s
I20240502 14:04:34.477921 12346 raft_consensus.cc:348] T b0d765fb4e1e49c2b654e9917e89c631 P a120eeb181c24820a8d17de6f1efd322 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } attrs { replace: true } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } }
I20240502 14:04:34.478665 12346 raft_consensus.cc:729] T b0d765fb4e1e49c2b654e9917e89c631 P a120eeb181c24820a8d17de6f1efd322 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a120eeb181c24820a8d17de6f1efd322, State: Initialized, Role: LEARNER
I20240502 14:04:34.479171 12346 consensus_queue.cc:260] T b0d765fb4e1e49c2b654e9917e89c631 P a120eeb181c24820a8d17de6f1efd322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } attrs { replace: true } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } }
I20240502 14:04:34.482285 12346 ts_tablet_manager.cc:1398] T b0d765fb4e1e49c2b654e9917e89c631 P a120eeb181c24820a8d17de6f1efd322: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240502 14:04:34.484560 12354 ts_tablet_manager.cc:902] T 5509e46f7fc94f32b0f59c7063e87546 P a120eeb181c24820a8d17de6f1efd322: Initiating tablet copy from peer 80949093e9ab44a8ad5f62462d0d5414 (127.11.9.67:39977)
I20240502 14:04:34.486852 12128 tablet_copy_service.cc:342] P 80949093e9ab44a8ad5f62462d0d5414: Request end of tablet copy session a120eeb181c24820a8d17de6f1efd322-b0d765fb4e1e49c2b654e9917e89c631 received from {username='slave'} at 127.0.0.1:43968
I20240502 14:04:34.487437 12128 tablet_copy_service.cc:433] P 80949093e9ab44a8ad5f62462d0d5414: ending tablet copy session a120eeb181c24820a8d17de6f1efd322-b0d765fb4e1e49c2b654e9917e89c631 on tablet b0d765fb4e1e49c2b654e9917e89c631 with peer a120eeb181c24820a8d17de6f1efd322
I20240502 14:04:34.496321 12354 tablet_copy_client.cc:308] T 5509e46f7fc94f32b0f59c7063e87546 P a120eeb181c24820a8d17de6f1efd322: tablet copy: Beginning tablet copy session from remote peer at address 127.11.9.67:39977
I20240502 14:04:34.497910 12128 tablet_copy_service.cc:140] P 80949093e9ab44a8ad5f62462d0d5414: Received BeginTabletCopySession request for tablet 5509e46f7fc94f32b0f59c7063e87546 from peer a120eeb181c24820a8d17de6f1efd322 ({username='slave'} at 127.0.0.1:43968)
I20240502 14:04:34.498411 12128 tablet_copy_service.cc:161] P 80949093e9ab44a8ad5f62462d0d5414: Beginning new tablet copy session on tablet 5509e46f7fc94f32b0f59c7063e87546 from peer a120eeb181c24820a8d17de6f1efd322 at {username='slave'} at 127.0.0.1:43968: session id = a120eeb181c24820a8d17de6f1efd322-5509e46f7fc94f32b0f59c7063e87546
I20240502 14:04:34.503710 12128 tablet_copy_source_session.cc:205] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:04:34.506326 12354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5509e46f7fc94f32b0f59c7063e87546. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:34.517993 12354 tablet_copy_client.cc:783] T 5509e46f7fc94f32b0f59c7063e87546 P a120eeb181c24820a8d17de6f1efd322: tablet copy: Starting download of 0 data blocks...
I20240502 14:04:34.518508 12354 tablet_copy_client.cc:647] T 5509e46f7fc94f32b0f59c7063e87546 P a120eeb181c24820a8d17de6f1efd322: tablet copy: Starting download of 1 WAL segments...
I20240502 14:04:34.521658 12354 tablet_copy_client.cc:523] T 5509e46f7fc94f32b0f59c7063e87546 P a120eeb181c24820a8d17de6f1efd322: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:04:34.527240 12354 tablet_bootstrap.cc:492] T 5509e46f7fc94f32b0f59c7063e87546 P a120eeb181c24820a8d17de6f1efd322: Bootstrap starting.
I20240502 14:04:34.542522 12354 tablet_bootstrap.cc:492] T 5509e46f7fc94f32b0f59c7063e87546 P a120eeb181c24820a8d17de6f1efd322: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:04:34.543144 12354 tablet_bootstrap.cc:492] T 5509e46f7fc94f32b0f59c7063e87546 P a120eeb181c24820a8d17de6f1efd322: Bootstrap complete.
I20240502 14:04:34.543562 12354 ts_tablet_manager.cc:1367] T 5509e46f7fc94f32b0f59c7063e87546 P a120eeb181c24820a8d17de6f1efd322: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.006s
I20240502 14:04:34.545611 12354 raft_consensus.cc:348] T 5509e46f7fc94f32b0f59c7063e87546 P a120eeb181c24820a8d17de6f1efd322 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } attrs { replace: true } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } }
I20240502 14:04:34.546394 12354 raft_consensus.cc:729] T 5509e46f7fc94f32b0f59c7063e87546 P a120eeb181c24820a8d17de6f1efd322 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a120eeb181c24820a8d17de6f1efd322, State: Initialized, Role: LEARNER
I20240502 14:04:34.547000 12354 consensus_queue.cc:260] T 5509e46f7fc94f32b0f59c7063e87546 P a120eeb181c24820a8d17de6f1efd322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } attrs { replace: true } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } }
I20240502 14:04:34.549050 12354 ts_tablet_manager.cc:1398] T 5509e46f7fc94f32b0f59c7063e87546 P a120eeb181c24820a8d17de6f1efd322: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:04:34.550479 12128 tablet_copy_service.cc:342] P 80949093e9ab44a8ad5f62462d0d5414: Request end of tablet copy session a120eeb181c24820a8d17de6f1efd322-5509e46f7fc94f32b0f59c7063e87546 received from {username='slave'} at 127.0.0.1:43968
I20240502 14:04:34.550850 12128 tablet_copy_service.cc:433] P 80949093e9ab44a8ad5f62462d0d5414: ending tablet copy session a120eeb181c24820a8d17de6f1efd322-5509e46f7fc94f32b0f59c7063e87546 on tablet 5509e46f7fc94f32b0f59c7063e87546 with peer a120eeb181c24820a8d17de6f1efd322
I20240502 14:04:34.619418 12354 ts_tablet_manager.cc:902] T 7284f9efdc5745c1a8f63af14c24d838 P a120eeb181c24820a8d17de6f1efd322: Initiating tablet copy from peer 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197)
I20240502 14:04:34.624835 12354 tablet_copy_client.cc:308] T 7284f9efdc5745c1a8f63af14c24d838 P a120eeb181c24820a8d17de6f1efd322: tablet copy: Beginning tablet copy session from remote peer at address 127.11.9.65:45197
I20240502 14:04:34.634330 11979 tablet_copy_service.cc:140] P 5c6d12fd1dd9431e9caac1ca05c3640f: Received BeginTabletCopySession request for tablet 7284f9efdc5745c1a8f63af14c24d838 from peer a120eeb181c24820a8d17de6f1efd322 ({username='slave'} at 127.0.0.1:42362)
I20240502 14:04:34.634738 11979 tablet_copy_service.cc:161] P 5c6d12fd1dd9431e9caac1ca05c3640f: Beginning new tablet copy session on tablet 7284f9efdc5745c1a8f63af14c24d838 from peer a120eeb181c24820a8d17de6f1efd322 at {username='slave'} at 127.0.0.1:42362: session id = a120eeb181c24820a8d17de6f1efd322-7284f9efdc5745c1a8f63af14c24d838
I20240502 14:04:34.640755 11979 tablet_copy_source_session.cc:205] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:04:34.642904 12354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7284f9efdc5745c1a8f63af14c24d838. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:34.653365 12354 tablet_copy_client.cc:783] T 7284f9efdc5745c1a8f63af14c24d838 P a120eeb181c24820a8d17de6f1efd322: tablet copy: Starting download of 0 data blocks...
I20240502 14:04:34.653895 12354 tablet_copy_client.cc:647] T 7284f9efdc5745c1a8f63af14c24d838 P a120eeb181c24820a8d17de6f1efd322: tablet copy: Starting download of 1 WAL segments...
I20240502 14:04:34.656769 12354 tablet_copy_client.cc:523] T 7284f9efdc5745c1a8f63af14c24d838 P a120eeb181c24820a8d17de6f1efd322: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:04:34.663717 12354 tablet_bootstrap.cc:492] T 7284f9efdc5745c1a8f63af14c24d838 P a120eeb181c24820a8d17de6f1efd322: Bootstrap starting.
I20240502 14:04:34.679314 12354 tablet_bootstrap.cc:492] T 7284f9efdc5745c1a8f63af14c24d838 P a120eeb181c24820a8d17de6f1efd322: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:04:34.680150 12354 tablet_bootstrap.cc:492] T 7284f9efdc5745c1a8f63af14c24d838 P a120eeb181c24820a8d17de6f1efd322: Bootstrap complete.
I20240502 14:04:34.680761 12354 ts_tablet_manager.cc:1367] T 7284f9efdc5745c1a8f63af14c24d838 P a120eeb181c24820a8d17de6f1efd322: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20240502 14:04:34.682626 12346 ts_tablet_manager.cc:902] T 3f82d6f517dc44f18f6d054cd4590124 P a120eeb181c24820a8d17de6f1efd322: Initiating tablet copy from peer 80949093e9ab44a8ad5f62462d0d5414 (127.11.9.67:39977)
I20240502 14:04:34.683588 12354 raft_consensus.cc:348] T 7284f9efdc5745c1a8f63af14c24d838 P a120eeb181c24820a8d17de6f1efd322 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } attrs { replace: true } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } }
I20240502 14:04:34.684479 12354 raft_consensus.cc:729] T 7284f9efdc5745c1a8f63af14c24d838 P a120eeb181c24820a8d17de6f1efd322 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a120eeb181c24820a8d17de6f1efd322, State: Initialized, Role: LEARNER
I20240502 14:04:34.685171 12354 consensus_queue.cc:260] T 7284f9efdc5745c1a8f63af14c24d838 P a120eeb181c24820a8d17de6f1efd322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } attrs { replace: true } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } }
I20240502 14:04:34.687175 12354 ts_tablet_manager.cc:1398] T 7284f9efdc5745c1a8f63af14c24d838 P a120eeb181c24820a8d17de6f1efd322: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20240502 14:04:34.688762 11979 tablet_copy_service.cc:342] P 5c6d12fd1dd9431e9caac1ca05c3640f: Request end of tablet copy session a120eeb181c24820a8d17de6f1efd322-7284f9efdc5745c1a8f63af14c24d838 received from {username='slave'} at 127.0.0.1:42362
I20240502 14:04:34.689116 11979 tablet_copy_service.cc:433] P 5c6d12fd1dd9431e9caac1ca05c3640f: ending tablet copy session a120eeb181c24820a8d17de6f1efd322-7284f9efdc5745c1a8f63af14c24d838 on tablet 7284f9efdc5745c1a8f63af14c24d838 with peer a120eeb181c24820a8d17de6f1efd322
I20240502 14:04:34.694049 12346 tablet_copy_client.cc:308] T 3f82d6f517dc44f18f6d054cd4590124 P a120eeb181c24820a8d17de6f1efd322: tablet copy: Beginning tablet copy session from remote peer at address 127.11.9.67:39977
I20240502 14:04:34.695466 12128 tablet_copy_service.cc:140] P 80949093e9ab44a8ad5f62462d0d5414: Received BeginTabletCopySession request for tablet 3f82d6f517dc44f18f6d054cd4590124 from peer a120eeb181c24820a8d17de6f1efd322 ({username='slave'} at 127.0.0.1:43968)
I20240502 14:04:34.695971 12128 tablet_copy_service.cc:161] P 80949093e9ab44a8ad5f62462d0d5414: Beginning new tablet copy session on tablet 3f82d6f517dc44f18f6d054cd4590124 from peer a120eeb181c24820a8d17de6f1efd322 at {username='slave'} at 127.0.0.1:43968: session id = a120eeb181c24820a8d17de6f1efd322-3f82d6f517dc44f18f6d054cd4590124
I20240502 14:04:34.701506 12128 tablet_copy_source_session.cc:205] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:04:34.704017 12346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f82d6f517dc44f18f6d054cd4590124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:34.714988 12346 tablet_copy_client.cc:783] T 3f82d6f517dc44f18f6d054cd4590124 P a120eeb181c24820a8d17de6f1efd322: tablet copy: Starting download of 0 data blocks...
I20240502 14:04:34.715478 12346 tablet_copy_client.cc:647] T 3f82d6f517dc44f18f6d054cd4590124 P a120eeb181c24820a8d17de6f1efd322: tablet copy: Starting download of 1 WAL segments...
I20240502 14:04:34.718403 12346 tablet_copy_client.cc:523] T 3f82d6f517dc44f18f6d054cd4590124 P a120eeb181c24820a8d17de6f1efd322: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:04:34.724140 12346 tablet_bootstrap.cc:492] T 3f82d6f517dc44f18f6d054cd4590124 P a120eeb181c24820a8d17de6f1efd322: Bootstrap starting.
I20240502 14:04:34.738260 12346 tablet_bootstrap.cc:492] T 3f82d6f517dc44f18f6d054cd4590124 P a120eeb181c24820a8d17de6f1efd322: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:04:34.738878 12346 tablet_bootstrap.cc:492] T 3f82d6f517dc44f18f6d054cd4590124 P a120eeb181c24820a8d17de6f1efd322: Bootstrap complete.
I20240502 14:04:34.739521 12346 ts_tablet_manager.cc:1367] T 3f82d6f517dc44f18f6d054cd4590124 P a120eeb181c24820a8d17de6f1efd322: Time spent bootstrapping tablet: real 0.016s user 0.015s sys 0.001s
I20240502 14:04:34.741814 12346 raft_consensus.cc:348] T 3f82d6f517dc44f18f6d054cd4590124 P a120eeb181c24820a8d17de6f1efd322 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } attrs { replace: true } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } }
I20240502 14:04:34.742462 12346 raft_consensus.cc:729] T 3f82d6f517dc44f18f6d054cd4590124 P a120eeb181c24820a8d17de6f1efd322 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a120eeb181c24820a8d17de6f1efd322, State: Initialized, Role: LEARNER
I20240502 14:04:34.743022 12346 consensus_queue.cc:260] T 3f82d6f517dc44f18f6d054cd4590124 P a120eeb181c24820a8d17de6f1efd322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5c6d12fd1dd9431e9caac1ca05c3640f" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 45197 } attrs { replace: true } } peers { permanent_uuid: "f12a3c49abe14896b2803edbd5662e42" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 40353 } } peers { permanent_uuid: "80949093e9ab44a8ad5f62462d0d5414" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 39977 } } peers { permanent_uuid: "a120eeb181c24820a8d17de6f1efd322" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39447 } attrs { promote: true } }
I20240502 14:04:34.744915 12346 ts_tablet_manager.cc:1398] T 3f82d6f517dc44f18f6d054cd4590124 P a120eeb181c24820a8d17de6f1efd322: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:04:34.746459 12128 tablet_copy_service.cc:342] P 80949093e9ab44a8ad5f62462d0d5414: Request end of tablet copy session a120eeb181c24820a8d17de6f1efd322-3f82d6f517dc44f18f6d054cd4590124 received from {username='slave'} at 127.0.0.1:43968
I20240502 14:04:34.746836 12128 tablet_copy_service.cc:433] P 80949093e9ab44a8ad5f62462d0d5414: ending tablet copy session a120eeb181c24820a8d17de6f1efd322-3f82d6f517dc44f18f6d054cd4590124 on tablet 3f82d6f517dc44f18f6d054cd4590124 with peer a120eeb181c24820a8d17de6f1efd322
I20240502 14:04:34.861994 12282 raft_consensus.cc:1203] T b0d765fb4e1e49c2b654e9917e89c631 P a120eeb181c24820a8d17de6f1efd322 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:04:34.915190 12282 raft_consensus.cc:1203] T 5509e46f7fc94f32b0f59c7063e87546 P a120eeb181c24820a8d17de6f1efd322 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:04:34.980423 11920 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:04:34.987394 11920 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:04:34.987821 11920 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:04:35.028079 11301 tablet_server.cc:178] TabletServer@127.11.9.65:0 shutting down...
W20240502 14:04:35.038645 11853 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 5 similar messages]
W20240502 14:04:35.043030 11915 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 14:04:35.043386 11915 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 14:04:35.061364 12009 consensus_peers.cc:485] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42 -> Peer 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197): Couldn't send request to peer 5c6d12fd1dd9431e9caac1ca05c3640f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:04:35.066159 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:04:35.066872 11301 tablet_replica.cc:330] T 3f82d6f517dc44f18f6d054cd4590124 P 5c6d12fd1dd9431e9caac1ca05c3640f: stopping tablet replica
I20240502 14:04:35.067746 11301 raft_consensus.cc:2229] T 3f82d6f517dc44f18f6d054cd4590124 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:35.068292 11301 raft_consensus.cc:2258] T 3f82d6f517dc44f18f6d054cd4590124 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:35.070739 11301 tablet_replica.cc:330] T b0d765fb4e1e49c2b654e9917e89c631 P 5c6d12fd1dd9431e9caac1ca05c3640f: stopping tablet replica
I20240502 14:04:35.071404 11301 raft_consensus.cc:2229] T b0d765fb4e1e49c2b654e9917e89c631 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:35.071897 11301 raft_consensus.cc:2258] T b0d765fb4e1e49c2b654e9917e89c631 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:35.074023 11301 tablet_replica.cc:330] T 61aee16dda4d415c85536390284665d9 P 5c6d12fd1dd9431e9caac1ca05c3640f: stopping tablet replica
I20240502 14:04:35.074704 11301 raft_consensus.cc:2229] T 61aee16dda4d415c85536390284665d9 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:35.075142 11301 raft_consensus.cc:2258] T 61aee16dda4d415c85536390284665d9 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:35.077371 11301 tablet_replica.cc:330] T 39b0b3c587fa44868a9b1175a2bba670 P 5c6d12fd1dd9431e9caac1ca05c3640f: stopping tablet replica
I20240502 14:04:35.077877 11301 raft_consensus.cc:2229] T 39b0b3c587fa44868a9b1175a2bba670 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:35.078351 11301 raft_consensus.cc:2258] T 39b0b3c587fa44868a9b1175a2bba670 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:35.080021 11301 tablet_replica.cc:330] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f: stopping tablet replica
I20240502 14:04:35.080462 11301 raft_consensus.cc:2229] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:04:35.081178 11301 raft_consensus.cc:2258] T 3c17b90e024548eb89f83fa426c08401 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:35.082648 11301 tablet_replica.cc:330] T 5509e46f7fc94f32b0f59c7063e87546 P 5c6d12fd1dd9431e9caac1ca05c3640f: stopping tablet replica
I20240502 14:04:35.083148 11301 raft_consensus.cc:2229] T 5509e46f7fc94f32b0f59c7063e87546 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:35.083563 11301 raft_consensus.cc:2258] T 5509e46f7fc94f32b0f59c7063e87546 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:35.085111 11301 tablet_replica.cc:330] T fb15de48a4294823b60b2892a4241ad0 P 5c6d12fd1dd9431e9caac1ca05c3640f: stopping tablet replica
I20240502 14:04:35.085680 11301 raft_consensus.cc:2229] T fb15de48a4294823b60b2892a4241ad0 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:35.086073 11301 raft_consensus.cc:2258] T fb15de48a4294823b60b2892a4241ad0 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:35.087764 11301 tablet_replica.cc:330] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f: stopping tablet replica
I20240502 14:04:35.088274 11301 raft_consensus.cc:2229] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:04:35.089076 11301 raft_consensus.cc:2258] T 7284f9efdc5745c1a8f63af14c24d838 P 5c6d12fd1dd9431e9caac1ca05c3640f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:35.099917 12282 raft_consensus.cc:1203] T 3f82d6f517dc44f18f6d054cd4590124 P a120eeb181c24820a8d17de6f1efd322 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240502 14:04:35.118257 12083 consensus_peers.cc:485] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 -> Peer 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197): Couldn't send request to peer 5c6d12fd1dd9431e9caac1ca05c3640f. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.65:45197: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:04:35.121116 11301 tablet_server.cc:194] TabletServer@127.11.9.65:0 shutdown complete.
I20240502 14:04:35.143000 11301 tablet_server.cc:178] TabletServer@127.11.9.66:0 shutting down...
W20240502 14:04:35.170325 12009 consensus_peers.cc:485] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42 -> Peer 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197): Couldn't send request to peer 5c6d12fd1dd9431e9caac1ca05c3640f. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.65:45197: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:04:35.186959 12083 consensus_peers.cc:485] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 -> Peer 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197): Couldn't send request to peer 5c6d12fd1dd9431e9caac1ca05c3640f. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.65:45197: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:04:35.196244 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:04:35.197121 11301 tablet_replica.cc:330] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42: stopping tablet replica
I20240502 14:04:35.197901 11301 raft_consensus.cc:2229] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:04:35.198839 11301 raft_consensus.cc:2258] T fb15de48a4294823b60b2892a4241ad0 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:35.200875 11301 tablet_replica.cc:330] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42: stopping tablet replica
I20240502 14:04:35.201541 11301 raft_consensus.cc:2229] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:04:35.202253 11301 raft_consensus.cc:2258] T 39b0b3c587fa44868a9b1175a2bba670 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:35.204206 11301 tablet_replica.cc:330] T 3f82d6f517dc44f18f6d054cd4590124 P f12a3c49abe14896b2803edbd5662e42: stopping tablet replica
I20240502 14:04:35.204672 11301 raft_consensus.cc:2229] T 3f82d6f517dc44f18f6d054cd4590124 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:35.205058 11301 raft_consensus.cc:2258] T 3f82d6f517dc44f18f6d054cd4590124 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:35.206914 11301 tablet_replica.cc:330] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42: stopping tablet replica
I20240502 14:04:35.207358 11301 raft_consensus.cc:2229] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:04:35.207983 11301 raft_consensus.cc:2258] T 61aee16dda4d415c85536390284665d9 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:35.209856 11301 tablet_replica.cc:330] T b0d765fb4e1e49c2b654e9917e89c631 P f12a3c49abe14896b2803edbd5662e42: stopping tablet replica
I20240502 14:04:35.210393 11301 raft_consensus.cc:2229] T b0d765fb4e1e49c2b654e9917e89c631 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:35.210868 11301 raft_consensus.cc:2258] T b0d765fb4e1e49c2b654e9917e89c631 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:35.212834 11301 tablet_replica.cc:330] T 7284f9efdc5745c1a8f63af14c24d838 P f12a3c49abe14896b2803edbd5662e42: stopping tablet replica
I20240502 14:04:35.213398 11301 raft_consensus.cc:2229] T 7284f9efdc5745c1a8f63af14c24d838 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:35.213944 11301 raft_consensus.cc:2258] T 7284f9efdc5745c1a8f63af14c24d838 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:35.215615 11301 tablet_replica.cc:330] T 3c17b90e024548eb89f83fa426c08401 P f12a3c49abe14896b2803edbd5662e42: stopping tablet replica
I20240502 14:04:35.216280 11301 raft_consensus.cc:2229] T 3c17b90e024548eb89f83fa426c08401 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:35.216703 11301 raft_consensus.cc:2258] T 3c17b90e024548eb89f83fa426c08401 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:35.218504 11301 tablet_replica.cc:330] T 5509e46f7fc94f32b0f59c7063e87546 P f12a3c49abe14896b2803edbd5662e42: stopping tablet replica
I20240502 14:04:35.219038 11301 raft_consensus.cc:2229] T 5509e46f7fc94f32b0f59c7063e87546 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:35.219446 11301 raft_consensus.cc:2258] T 5509e46f7fc94f32b0f59c7063e87546 P f12a3c49abe14896b2803edbd5662e42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:35.237802 11301 tablet_server.cc:194] TabletServer@127.11.9.66:0 shutdown complete.
W20240502 14:04:35.246024 12083 consensus_peers.cc:485] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 -> Peer 5c6d12fd1dd9431e9caac1ca05c3640f (127.11.9.65:45197): Couldn't send request to peer 5c6d12fd1dd9431e9caac1ca05c3640f. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.65:45197: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:04:35.259555 11301 tablet_server.cc:178] TabletServer@127.11.9.67:0 shutting down...
W20240502 14:04:35.263983 11915 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 14:04:35.264182 11915 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 14:04:35.267160 11915 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 14:04:35.267477 11915 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 14:04:35.271725 11915 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 14:04:35.271936 11915 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240502 14:04:35.283164 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:04:35.283816 11301 tablet_replica.cc:330] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414: stopping tablet replica
I20240502 14:04:35.284407 11301 raft_consensus.cc:2229] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:04:35.285295 11301 raft_consensus.cc:2258] T b0d765fb4e1e49c2b654e9917e89c631 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:35.286902 11301 tablet_replica.cc:330] T fb15de48a4294823b60b2892a4241ad0 P 80949093e9ab44a8ad5f62462d0d5414: stopping tablet replica
I20240502 14:04:35.287353 11301 raft_consensus.cc:2229] T fb15de48a4294823b60b2892a4241ad0 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:35.287707 11301 raft_consensus.cc:2258] T fb15de48a4294823b60b2892a4241ad0 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:35.289129 11301 tablet_replica.cc:330] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414: stopping tablet replica
I20240502 14:04:35.289593 11301 raft_consensus.cc:2229] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:04:35.290385 11301 raft_consensus.cc:2258] T 3f82d6f517dc44f18f6d054cd4590124 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:35.291806 11301 tablet_replica.cc:330] T 61aee16dda4d415c85536390284665d9 P 80949093e9ab44a8ad5f62462d0d5414: stopping tablet replica
I20240502 14:04:35.292275 11301 raft_consensus.cc:2229] T 61aee16dda4d415c85536390284665d9 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:35.292594 11301 raft_consensus.cc:2258] T 61aee16dda4d415c85536390284665d9 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:35.294240 11301 tablet_replica.cc:330] T 7284f9efdc5745c1a8f63af14c24d838 P 80949093e9ab44a8ad5f62462d0d5414: stopping tablet replica
I20240502 14:04:35.294653 11301 raft_consensus.cc:2229] T 7284f9efdc5745c1a8f63af14c24d838 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:35.295038 11301 raft_consensus.cc:2258] T 7284f9efdc5745c1a8f63af14c24d838 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:35.296502 11301 tablet_replica.cc:330] T 39b0b3c587fa44868a9b1175a2bba670 P 80949093e9ab44a8ad5f62462d0d5414: stopping tablet replica
I20240502 14:04:35.296939 11301 raft_consensus.cc:2229] T 39b0b3c587fa44868a9b1175a2bba670 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:35.297276 11301 raft_consensus.cc:2258] T 39b0b3c587fa44868a9b1175a2bba670 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:35.298712 11301 tablet_replica.cc:330] T 3c17b90e024548eb89f83fa426c08401 P 80949093e9ab44a8ad5f62462d0d5414: stopping tablet replica
I20240502 14:04:35.299119 11301 raft_consensus.cc:2229] T 3c17b90e024548eb89f83fa426c08401 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:35.299466 11301 raft_consensus.cc:2258] T 3c17b90e024548eb89f83fa426c08401 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:35.300920 11301 tablet_replica.cc:330] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414: stopping tablet replica
I20240502 14:04:35.301353 11301 raft_consensus.cc:2229] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:04:35.302161 11301 raft_consensus.cc:2258] T 5509e46f7fc94f32b0f59c7063e87546 P 80949093e9ab44a8ad5f62462d0d5414 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:35.318470 11301 tablet_server.cc:194] TabletServer@127.11.9.67:0 shutdown complete.
I20240502 14:04:35.334616 11301 tablet_server.cc:178] TabletServer@127.11.9.68:0 shutting down...
I20240502 14:04:35.355670 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:04:35.356252 11301 tablet_replica.cc:330] T 3f82d6f517dc44f18f6d054cd4590124 P a120eeb181c24820a8d17de6f1efd322: stopping tablet replica
I20240502 14:04:35.356770 11301 raft_consensus.cc:2229] T 3f82d6f517dc44f18f6d054cd4590124 P a120eeb181c24820a8d17de6f1efd322 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 14:04:35.357133 11301 raft_consensus.cc:2258] T 3f82d6f517dc44f18f6d054cd4590124 P a120eeb181c24820a8d17de6f1efd322 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 14:04:35.358704 11301 tablet_replica.cc:330] T 7284f9efdc5745c1a8f63af14c24d838 P a120eeb181c24820a8d17de6f1efd322: stopping tablet replica
I20240502 14:04:35.359163 11301 raft_consensus.cc:2229] T 7284f9efdc5745c1a8f63af14c24d838 P a120eeb181c24820a8d17de6f1efd322 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 14:04:35.359476 11301 raft_consensus.cc:2258] T 7284f9efdc5745c1a8f63af14c24d838 P a120eeb181c24820a8d17de6f1efd322 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 14:04:35.360817 11301 tablet_replica.cc:330] T 5509e46f7fc94f32b0f59c7063e87546 P a120eeb181c24820a8d17de6f1efd322: stopping tablet replica
I20240502 14:04:35.361232 11301 raft_consensus.cc:2229] T 5509e46f7fc94f32b0f59c7063e87546 P a120eeb181c24820a8d17de6f1efd322 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 14:04:35.361585 11301 raft_consensus.cc:2258] T 5509e46f7fc94f32b0f59c7063e87546 P a120eeb181c24820a8d17de6f1efd322 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 14:04:35.363020 11301 tablet_replica.cc:330] T b0d765fb4e1e49c2b654e9917e89c631 P a120eeb181c24820a8d17de6f1efd322: stopping tablet replica
I20240502 14:04:35.363425 11301 raft_consensus.cc:2229] T b0d765fb4e1e49c2b654e9917e89c631 P a120eeb181c24820a8d17de6f1efd322 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 14:04:35.363731 11301 raft_consensus.cc:2258] T b0d765fb4e1e49c2b654e9917e89c631 P a120eeb181c24820a8d17de6f1efd322 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 14:04:35.383347 11301 tablet_server.cc:194] TabletServer@127.11.9.68:0 shutdown complete.
I20240502 14:04:35.392967 11301 master.cc:537] Master@127.11.9.126:36455 shutting down...
I20240502 14:04:35.410203 11301 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 11d49c87b2cb4ebb9f6d72955afe2cd0 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:04:35.410652 11301 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 11d49c87b2cb4ebb9f6d72955afe2cd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:35.410951 11301 tablet_replica.cc:330] T 00000000000000000000000000000000 P 11d49c87b2cb4ebb9f6d72955afe2cd0: stopping tablet replica
I20240502 14:04:35.444195 11301 master.cc:552] Master@127.11.9.126:36455 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (10690 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240502 14:04:35.484129 11301 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.9.126:35153,127.11.9.125:40629,127.11.9.124:45685
I20240502 14:04:35.485412 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:35.491322 12378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:35.491986 12379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:35.492641 12381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:35.493240 11301 server_base.cc:1034] running on GCE node
I20240502 14:04:35.494001 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:04:35.494189 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:04:35.494336 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658675494312 us; error 0 us; skew 500 ppm
I20240502 14:04:35.494798 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:04:35.497107 11301 webserver.cc:460] Webserver started at http://127.11.9.126:38145/ using document root <none> and password file <none>
I20240502 14:04:35.497577 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:35.497758 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:35.498032 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:35.499045 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-0-root/instance:
uuid: "0e1c6a2504b84982b5bb3cf32bff6e48"
format_stamp: "Formatted at 2024-05-02 14:04:35 on dist-test-slave-j72s"
I20240502 14:04:35.503183 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:04:35.506108 12386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:35.506881 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:04:35.507124 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-0-root
uuid: "0e1c6a2504b84982b5bb3cf32bff6e48"
format_stamp: "Formatted at 2024-05-02 14:04:35 on dist-test-slave-j72s"
I20240502 14:04:35.507360 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:35.527467 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:35.528573 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:35.564074 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.126:35153
I20240502 14:04:35.564224 12437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.126:35153 every 8 connection(s)
I20240502 14:04:35.567857 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 14:04:35.568110 12438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 14:04:35.573802 12440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:35.574816 12441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:35.574883 12438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:04:35.580262 11301 server_base.cc:1034] running on GCE node
W20240502 14:04:35.582589 12443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:35.583544 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:04:35.583808 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:04:35.584005 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658675583990 us; error 0 us; skew 500 ppm
I20240502 14:04:35.584478 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:04:35.586987 11301 webserver.cc:460] Webserver started at http://127.11.9.125:33115/ using document root <none> and password file <none>
I20240502 14:04:35.587433 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:35.587622 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:35.587981 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:35.589020 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-1-root/instance:
uuid: "3d445a06bca1419eaab962838f24c742"
format_stamp: "Formatted at 2024-05-02 14:04:35 on dist-test-slave-j72s"
I20240502 14:04:35.590994 12438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:04:35.593986 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
W20240502 14:04:35.595624 12438 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.11.9.125:40629: Network error: Client connection negotiation failed: client connection to 127.11.9.125:40629: connect: Connection refused (error 111)
I20240502 14:04:35.597489 12451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:35.598165 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:04:35.598429 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-1-root
uuid: "3d445a06bca1419eaab962838f24c742"
format_stamp: "Formatted at 2024-05-02 14:04:35 on dist-test-slave-j72s"
I20240502 14:04:35.598690 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:35.630190 12438 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } attempt: 1
I20240502 14:04:35.630828 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:35.632182 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
W20240502 14:04:35.634210 12438 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.11.9.125:40629: Network error: Client connection negotiation failed: client connection to 127.11.9.125:40629: connect: Connection refused (error 111)
I20240502 14:04:35.668429 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.125:40629
I20240502 14:04:35.668525 12502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.125:40629 every 8 connection(s)
I20240502 14:04:35.671887 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 14:04:35.672169 12503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 14:04:35.677814 12505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:35.678789 12506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:35.679488 12503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } has no permanent_uuid. Determining permanent_uuid...
W20240502 14:04:35.681813 12508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:35.683282 11301 server_base.cc:1034] running on GCE node
I20240502 14:04:35.684317 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:04:35.684590 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:04:35.684795 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658675684775 us; error 0 us; skew 500 ppm
I20240502 14:04:35.685348 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:04:35.688271 11301 webserver.cc:460] Webserver started at http://127.11.9.124:41225/ using document root <none> and password file <none>
I20240502 14:04:35.688730 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:35.688921 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:35.689175 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:35.690596 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-2-root/instance:
uuid: "91973bdfb46543e9b95f811246c6377f"
format_stamp: "Formatted at 2024-05-02 14:04:35 on dist-test-slave-j72s"
I20240502 14:04:35.691550 12503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:04:35.694700 12438 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } attempt: 2
I20240502 14:04:35.694973 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20240502 14:04:35.705487 12503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:04:35.705669 12516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:35.706526 12438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:04:35.706583 11301 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.005s
I20240502 14:04:35.707046 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-2-root
uuid: "91973bdfb46543e9b95f811246c6377f"
format_stamp: "Formatted at 2024-05-02 14:04:35 on dist-test-slave-j72s"
I20240502 14:04:35.707404 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240502 14:04:35.710714 12438 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.11.9.124:45685: Network error: Client connection negotiation failed: client connection to 127.11.9.124:45685: connect: Connection refused (error 111)
W20240502 14:04:35.711856 12503 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.11.9.124:45685: Network error: Client connection negotiation failed: client connection to 127.11.9.124:45685: connect: Connection refused (error 111)
I20240502 14:04:35.723372 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:35.724442 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:35.744251 12438 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } attempt: 1
W20240502 14:04:35.749115 12438 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.11.9.124:45685: Network error: Client connection negotiation failed: client connection to 127.11.9.124:45685: connect: Connection refused (error 111)
I20240502 14:04:35.760084 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.124:45685
I20240502 14:04:35.760179 12568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.124:45685 every 8 connection(s)
I20240502 14:04:35.762924 11301 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:04:35.763523 12569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:35.768808 12569 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:04:35.777344 12503 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } attempt: 1
I20240502 14:04:35.778137 12569 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:04:35.788172 12569 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } has no permanent_uuid. Determining permanent_uuid...
I20240502 14:04:35.792568 12438 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } attempt: 2
I20240502 14:04:35.794608 12503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742: Bootstrap starting.
I20240502 14:04:35.800040 12503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:35.802886 12569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f: Bootstrap starting.
I20240502 14:04:35.805472 12503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742: No bootstrap required, opened a new log
I20240502 14:04:35.807339 12503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } }
I20240502 14:04:35.807886 12503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:35.808105 12503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d445a06bca1419eaab962838f24c742, State: Initialized, Role: FOLLOWER
I20240502 14:04:35.808523 12569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:35.808732 12503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } }
I20240502 14:04:35.810642 12575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } } }
I20240502 14:04:35.811538 12575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:04:35.812142 12503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:04:35.812528 12438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48: Bootstrap starting.
I20240502 14:04:35.819949 12438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:35.825963 12438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48: No bootstrap required, opened a new log
I20240502 14:04:35.827040 12569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f: No bootstrap required, opened a new log
I20240502 14:04:35.828313 12438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } }
I20240502 14:04:35.829002 12438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:35.829311 12438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1c6a2504b84982b5bb3cf32bff6e48, State: Initialized, Role: FOLLOWER
I20240502 14:04:35.829490 12569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } }
I20240502 14:04:35.830298 12569 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:35.830623 12569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91973bdfb46543e9b95f811246c6377f, State: Initialized, Role: FOLLOWER
I20240502 14:04:35.830163 12438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } }
I20240502 14:04:35.831359 12569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } }
I20240502 14:04:35.832618 12589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } } }
W20240502 14:04:35.833220 12588 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 14:04:35.833557 12588 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 14:04:35.834421 12438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:04:35.834630 12590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } } }
I20240502 14:04:35.833431 12589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:04:35.835418 12590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:04:35.836120 12569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:04:35.849413 11301 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240502 14:04:35.849716 11301 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20240502 14:04:35.852144 12609 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 14:04:35.852416 12609 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 14:04:35.856501 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:35.858819 12612 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 14:04:35.859159 12612 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240502 14:04:35.864259 12613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:35.864986 12614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:35.865902 12616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:35.866351 11301 server_base.cc:1034] running on GCE node
I20240502 14:04:35.867023 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:04:35.867249 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:04:35.867437 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658675867411 us; error 0 us; skew 500 ppm
I20240502 14:04:35.867679 12590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:35.867906 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:04:35.868031 12590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } }
I20240502 14:04:35.869961 12590 leader_election.cc:288] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e1c6a2504b84982b5bb3cf32bff6e48 (127.11.9.126:35153), 3d445a06bca1419eaab962838f24c742 (127.11.9.125:40629)
I20240502 14:04:35.871192 11301 webserver.cc:460] Webserver started at http://127.11.9.65:40521/ using document root <none> and password file <none>
I20240502 14:04:35.871008 12478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91973bdfb46543e9b95f811246c6377f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d445a06bca1419eaab962838f24c742" is_pre_election: true
I20240502 14:04:35.870906 12413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91973bdfb46543e9b95f811246c6377f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" is_pre_election: true
I20240502 14:04:35.871762 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:35.871719 12478 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91973bdfb46543e9b95f811246c6377f in term 0.
I20240502 14:04:35.872182 12413 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91973bdfb46543e9b95f811246c6377f in term 0.
I20240502 14:04:35.872180 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:35.873307 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:35.872897 12518 leader_election.cc:302] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d445a06bca1419eaab962838f24c742, 91973bdfb46543e9b95f811246c6377f; no voters:
I20240502 14:04:35.875146 12590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:35.875203 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-0-root/instance:
uuid: "86c24be0852c4662926a2951c7e19394"
format_stamp: "Formatted at 2024-05-02 14:04:35 on dist-test-slave-j72s"
I20240502 14:04:35.875465 12590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:35.875824 12590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:35.880700 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 14:04:35.883831 12621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:35.884549 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 14:04:35.884829 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-0-root
uuid: "86c24be0852c4662926a2951c7e19394"
format_stamp: "Formatted at 2024-05-02 14:04:35 on dist-test-slave-j72s"
I20240502 14:04:35.885082 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:35.903736 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:35.904881 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:35.906250 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:04:35.908492 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:35.908682 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:35.908963 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:35.909144 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:35.910022 12590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } }
I20240502 14:04:35.912240 12590 leader_election.cc:288] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [CANDIDATE]: Term 1 election: Requested vote from peers 0e1c6a2504b84982b5bb3cf32bff6e48 (127.11.9.126:35153), 3d445a06bca1419eaab962838f24c742 (127.11.9.125:40629)
I20240502 14:04:35.913136 12413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91973bdfb46543e9b95f811246c6377f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48"
I20240502 14:04:35.913246 12478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91973bdfb46543e9b95f811246c6377f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d445a06bca1419eaab962838f24c742"
I20240502 14:04:35.913849 12413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:35.913908 12478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:35.948410 12413 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91973bdfb46543e9b95f811246c6377f in term 1.
I20240502 14:04:35.948428 12478 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91973bdfb46543e9b95f811246c6377f in term 1.
I20240502 14:04:35.949867 12519 leader_election.cc:302] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1c6a2504b84982b5bb3cf32bff6e48, 91973bdfb46543e9b95f811246c6377f; no voters:
I20240502 14:04:35.951038 12590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:35.952180 12590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [term 1 LEADER]: Becoming Leader. State: Replica: 91973bdfb46543e9b95f811246c6377f, State: Running, Role: LEADER
I20240502 14:04:35.952787 12590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } }
I20240502 14:04:35.956241 12685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.65:46227 every 8 connection(s)
I20240502 14:04:35.956832 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.65:46227
I20240502 14:04:35.958261 12682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91973bdfb46543e9b95f811246c6377f. Latest consensus state: current_term: 1 leader_uuid: "91973bdfb46543e9b95f811246c6377f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } } }
I20240502 14:04:35.959043 12682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [sys.catalog]: This master's current role is: LEADER
I20240502 14:04:35.960505 12686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:04:35.968276 12686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:04:35.986338 12478 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [term 1 FOLLOWER]: Refusing update from remote peer 91973bdfb46543e9b95f811246c6377f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:04:35.990193 12682 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:35.990528 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 14:04:35.995095 12687 heartbeater.cc:347] Connected to a master server at 127.11.9.124:45685
I20240502 14:04:35.995460 12687 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:35.996285 12687 heartbeater.cc:511] Master 127.11.9.124:45685 requested a full tablet report, sending...
I20240502 14:04:35.999487 12534 ts_manager.cc:194] Registered new tserver with Master: 86c24be0852c4662926a2951c7e19394 (127.11.9.65:46227)
I20240502 14:04:36.003098 12689 heartbeater.cc:347] Connected to a master server at 127.11.9.126:35153
I20240502 14:04:36.003582 12689 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:36.004683 12689 heartbeater.cc:511] Master 127.11.9.126:35153 requested a full tablet report, sending...
I20240502 14:04:36.004825 12413 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [term 1 FOLLOWER]: Refusing update from remote peer 91973bdfb46543e9b95f811246c6377f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:04:36.008800 12403 ts_manager.cc:194] Registered new tserver with Master: 86c24be0852c4662926a2951c7e19394 (127.11.9.65:46227)
I20240502 14:04:36.008873 12682 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240502 14:04:36.017179 12696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:36.018393 12688 heartbeater.cc:347] Connected to a master server at 127.11.9.125:40629
I20240502 14:04:36.018977 12688 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:36.020046 12688 heartbeater.cc:511] Master 127.11.9.125:40629 requested a full tablet report, sending...
I20240502 14:04:36.029134 12575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91973bdfb46543e9b95f811246c6377f. Latest consensus state: current_term: 1 leader_uuid: "91973bdfb46543e9b95f811246c6377f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } } }
I20240502 14:04:36.029750 12575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:04:36.028721 12589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91973bdfb46543e9b95f811246c6377f. Latest consensus state: current_term: 1 leader_uuid: "91973bdfb46543e9b95f811246c6377f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } } }
I20240502 14:04:36.030351 12589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [sys.catalog]: This master's current role is: FOLLOWER
W20240502 14:04:36.031407 12697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:36.031677 12468 ts_manager.cc:194] Registered new tserver with Master: 86c24be0852c4662926a2951c7e19394 (127.11.9.65:46227)
I20240502 14:04:36.038439 12590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "91973bdfb46543e9b95f811246c6377f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } } }
I20240502 14:04:36.039217 12590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [sys.catalog]: This master's current role is: LEADER
I20240502 14:04:36.039355 11301 server_base.cc:1034] running on GCE node
I20240502 14:04:36.040943 12686 catalog_manager.cc:1347] Generated new cluster ID: d5cabb8e01cc4ca98e4bac6421bea4be
I20240502 14:04:36.041249 12686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
W20240502 14:04:36.048269 12702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:36.040462 12590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "91973bdfb46543e9b95f811246c6377f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } } }
I20240502 14:04:36.048935 12590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [sys.catalog]: This master's current role is: LEADER
I20240502 14:04:36.049647 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:04:36.050025 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:04:36.050331 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658676050309 us; error 0 us; skew 500 ppm
I20240502 14:04:36.050156 12589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "91973bdfb46543e9b95f811246c6377f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } } }
I20240502 14:04:36.050891 12589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:04:36.051334 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:04:36.055812 12575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "91973bdfb46543e9b95f811246c6377f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } } }
I20240502 14:04:36.057694 12575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:04:36.058826 11301 webserver.cc:460] Webserver started at http://127.11.9.66:46485/ using document root <none> and password file <none>
I20240502 14:04:36.059298 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:36.059500 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:36.059733 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:36.060721 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-1-root/instance:
uuid: "ec0283e4f52e4572a67722ccc1097dad"
format_stamp: "Formatted at 2024-05-02 14:04:36 on dist-test-slave-j72s"
I20240502 14:04:36.066594 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 14:04:36.071682 12711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:36.072611 11301 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20240502 14:04:36.072916 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-1-root
uuid: "ec0283e4f52e4572a67722ccc1097dad"
format_stamp: "Formatted at 2024-05-02 14:04:36 on dist-test-slave-j72s"
I20240502 14:04:36.074466 12686 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:04:36.073246 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:36.075678 12686 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:04:36.096282 12686 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f: Generated new TSK 0
I20240502 14:04:36.097963 12686 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:04:36.104880 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:36.106103 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:36.107249 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:04:36.109568 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:36.109789 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:36.109959 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:36.110077 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:36.148598 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.66:46749
I20240502 14:04:36.148692 12773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.66:46749 every 8 connection(s)
I20240502 14:04:36.174930 12774 heartbeater.cc:347] Connected to a master server at 127.11.9.124:45685
I20240502 14:04:36.175499 12774 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:36.176528 12774 heartbeater.cc:511] Master 127.11.9.124:45685 requested a full tablet report, sending...
I20240502 14:04:36.176507 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 14:04:36.178815 12534 ts_manager.cc:194] Registered new tserver with Master: ec0283e4f52e4572a67722ccc1097dad (127.11.9.66:46749)
I20240502 14:04:36.180920 12775 heartbeater.cc:347] Connected to a master server at 127.11.9.125:40629
I20240502 14:04:36.181304 12775 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:36.181424 12776 heartbeater.cc:347] Connected to a master server at 127.11.9.126:35153
I20240502 14:04:36.181793 12534 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50834
I20240502 14:04:36.181893 12776 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:36.182839 12776 heartbeater.cc:511] Master 127.11.9.126:35153 requested a full tablet report, sending...
I20240502 14:04:36.183898 12775 heartbeater.cc:511] Master 127.11.9.125:40629 requested a full tablet report, sending...
I20240502 14:04:36.186769 12403 ts_manager.cc:194] Registered new tserver with Master: ec0283e4f52e4572a67722ccc1097dad (127.11.9.66:46749)
I20240502 14:04:36.187350 12468 ts_manager.cc:194] Registered new tserver with Master: ec0283e4f52e4572a67722ccc1097dad (127.11.9.66:46749)
W20240502 14:04:36.190968 12781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:36.191828 12782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:36.193564 12784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:36.194406 11301 server_base.cc:1034] running on GCE node
I20240502 14:04:36.195118 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:04:36.195335 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:04:36.195504 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658676195486 us; error 0 us; skew 500 ppm
I20240502 14:04:36.195976 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:04:36.198309 11301 webserver.cc:460] Webserver started at http://127.11.9.67:39833/ using document root <none> and password file <none>
I20240502 14:04:36.198807 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:36.199023 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:36.199267 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:36.200336 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-2-root/instance:
uuid: "688d5aecbeb74735aecb64a747dee84e"
format_stamp: "Formatted at 2024-05-02 14:04:36 on dist-test-slave-j72s"
I20240502 14:04:36.204408 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 14:04:36.207226 12789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:36.207938 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:04:36.208216 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-2-root
uuid: "688d5aecbeb74735aecb64a747dee84e"
format_stamp: "Formatted at 2024-05-02 14:04:36 on dist-test-slave-j72s"
I20240502 14:04:36.208537 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714658659912502-11301-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:36.225826 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:36.226805 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:36.228039 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:04:36.230113 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:36.230306 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:36.230540 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:36.230679 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:36.268095 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.67:35101
I20240502 14:04:36.268174 12851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.67:35101 every 8 connection(s)
I20240502 14:04:36.294742 12852 heartbeater.cc:347] Connected to a master server at 127.11.9.124:45685
I20240502 14:04:36.295233 12852 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:36.296165 12852 heartbeater.cc:511] Master 127.11.9.124:45685 requested a full tablet report, sending...
I20240502 14:04:36.298317 12853 heartbeater.cc:347] Connected to a master server at 127.11.9.125:40629
I20240502 14:04:36.298485 12534 ts_manager.cc:194] Registered new tserver with Master: 688d5aecbeb74735aecb64a747dee84e (127.11.9.67:35101)
I20240502 14:04:36.298700 12853 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:36.299479 12853 heartbeater.cc:511] Master 127.11.9.125:40629 requested a full tablet report, sending...
I20240502 14:04:36.300204 12534 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50850
I20240502 14:04:36.301936 12468 ts_manager.cc:194] Registered new tserver with Master: 688d5aecbeb74735aecb64a747dee84e (127.11.9.67:35101)
I20240502 14:04:36.303380 12854 heartbeater.cc:347] Connected to a master server at 127.11.9.126:35153
I20240502 14:04:36.303645 12854 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:36.304065 12854 heartbeater.cc:511] Master 127.11.9.126:35153 requested a full tablet report, sending...
I20240502 14:04:36.306259 12403 ts_manager.cc:194] Registered new tserver with Master: 688d5aecbeb74735aecb64a747dee84e (127.11.9.67:35101)
I20240502 14:04:36.307052 11301 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017560256s
I20240502 14:04:36.837025 12588 catalog_manager.cc:1259] Loaded cluster ID: d5cabb8e01cc4ca98e4bac6421bea4be
I20240502 14:04:36.837286 12588 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742: loading cluster ID for follower catalog manager: success
I20240502 14:04:36.841425 12588 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742: acquiring CA information for follower catalog manager: success
I20240502 14:04:36.844482 12588 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 14:04:36.855705 12609 catalog_manager.cc:1259] Loaded cluster ID: d5cabb8e01cc4ca98e4bac6421bea4be
I20240502 14:04:36.855966 12609 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48: loading cluster ID for follower catalog manager: success
I20240502 14:04:36.859898 12609 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48: acquiring CA information for follower catalog manager: success
I20240502 14:04:36.862648 12609 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 14:04:37.004200 12534 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50832
I20240502 14:04:37.188308 12774 heartbeater.cc:503] Master 127.11.9.124:45685 was elected leader, sending a full tablet report...
I20240502 14:04:37.302778 12852 heartbeater.cc:503] Master 127.11.9.124:45685 was elected leader, sending a full tablet report...
I20240502 14:04:37.854784 12611 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:04:37.870114 11301 test_util.cc:268] Using random seed: 359124492
I20240502 14:04:37.906978 12534 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50864:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 14:04:37.996764 12649 tablet_service.cc:1467] Processing CreateTablet for tablet dec6810c74994cb2b031154d236a6047 (DEFAULT_TABLE table=test-workload [id=735f9b44aa674bfdbe05ad3cc83f3a46]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240502 14:04:37.998162 12649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dec6810c74994cb2b031154d236a6047. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:37.998128 12817 tablet_service.cc:1467] Processing CreateTablet for tablet 571e065c13df4042a262adca43c13541 (DEFAULT_TABLE table=test-workload [id=735f9b44aa674bfdbe05ad3cc83f3a46]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240502 14:04:37.999382 12817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 571e065c13df4042a262adca43c13541. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:37.999630 12739 tablet_service.cc:1467] Processing CreateTablet for tablet b8d9fc8406104e59b6677c5eb048d950 (DEFAULT_TABLE table=test-workload [id=735f9b44aa674bfdbe05ad3cc83f3a46]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240502 14:04:38.000903 12739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8d9fc8406104e59b6677c5eb048d950. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:38.007071 12687 heartbeater.cc:503] Master 127.11.9.124:45685 was elected leader, sending a full tablet report...
I20240502 14:04:38.012405 12888 tablet_bootstrap.cc:492] T 571e065c13df4042a262adca43c13541 P 688d5aecbeb74735aecb64a747dee84e: Bootstrap starting.
I20240502 14:04:38.019534 12888 tablet_bootstrap.cc:654] T 571e065c13df4042a262adca43c13541 P 688d5aecbeb74735aecb64a747dee84e: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:38.022557 12890 tablet_bootstrap.cc:492] T b8d9fc8406104e59b6677c5eb048d950 P ec0283e4f52e4572a67722ccc1097dad: Bootstrap starting.
I20240502 14:04:38.022612 12889 tablet_bootstrap.cc:492] T dec6810c74994cb2b031154d236a6047 P 86c24be0852c4662926a2951c7e19394: Bootstrap starting.
I20240502 14:04:38.027767 12889 tablet_bootstrap.cc:654] T dec6810c74994cb2b031154d236a6047 P 86c24be0852c4662926a2951c7e19394: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:38.028098 12888 tablet_bootstrap.cc:492] T 571e065c13df4042a262adca43c13541 P 688d5aecbeb74735aecb64a747dee84e: No bootstrap required, opened a new log
I20240502 14:04:38.028568 12888 ts_tablet_manager.cc:1367] T 571e065c13df4042a262adca43c13541 P 688d5aecbeb74735aecb64a747dee84e: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20240502 14:04:38.029109 12890 tablet_bootstrap.cc:654] T b8d9fc8406104e59b6677c5eb048d950 P ec0283e4f52e4572a67722ccc1097dad: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:38.031337 12888 raft_consensus.cc:348] T 571e065c13df4042a262adca43c13541 P 688d5aecbeb74735aecb64a747dee84e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688d5aecbeb74735aecb64a747dee84e" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35101 } }
I20240502 14:04:38.032035 12889 tablet_bootstrap.cc:492] T dec6810c74994cb2b031154d236a6047 P 86c24be0852c4662926a2951c7e19394: No bootstrap required, opened a new log
I20240502 14:04:38.031971 12888 raft_consensus.cc:374] T 571e065c13df4042a262adca43c13541 P 688d5aecbeb74735aecb64a747dee84e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:38.032610 12888 raft_consensus.cc:729] T 571e065c13df4042a262adca43c13541 P 688d5aecbeb74735aecb64a747dee84e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 688d5aecbeb74735aecb64a747dee84e, State: Initialized, Role: FOLLOWER
I20240502 14:04:38.032667 12889 ts_tablet_manager.cc:1367] T dec6810c74994cb2b031154d236a6047 P 86c24be0852c4662926a2951c7e19394: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240502 14:04:38.033360 12888 consensus_queue.cc:260] T 571e065c13df4042a262adca43c13541 P 688d5aecbeb74735aecb64a747dee84e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688d5aecbeb74735aecb64a747dee84e" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35101 } }
I20240502 14:04:38.034081 12888 raft_consensus.cc:388] T 571e065c13df4042a262adca43c13541 P 688d5aecbeb74735aecb64a747dee84e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:04:38.034385 12888 raft_consensus.cc:482] T 571e065c13df4042a262adca43c13541 P 688d5aecbeb74735aecb64a747dee84e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:04:38.034729 12888 raft_consensus.cc:3037] T 571e065c13df4042a262adca43c13541 P 688d5aecbeb74735aecb64a747dee84e [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:38.034735 12890 tablet_bootstrap.cc:492] T b8d9fc8406104e59b6677c5eb048d950 P ec0283e4f52e4572a67722ccc1097dad: No bootstrap required, opened a new log
I20240502 14:04:38.034807 12889 raft_consensus.cc:348] T dec6810c74994cb2b031154d236a6047 P 86c24be0852c4662926a2951c7e19394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c24be0852c4662926a2951c7e19394" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 46227 } }
I20240502 14:04:38.035581 12889 raft_consensus.cc:374] T dec6810c74994cb2b031154d236a6047 P 86c24be0852c4662926a2951c7e19394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:38.035557 12890 ts_tablet_manager.cc:1367] T b8d9fc8406104e59b6677c5eb048d950 P ec0283e4f52e4572a67722ccc1097dad: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.006s
I20240502 14:04:38.036002 12889 raft_consensus.cc:729] T dec6810c74994cb2b031154d236a6047 P 86c24be0852c4662926a2951c7e19394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86c24be0852c4662926a2951c7e19394, State: Initialized, Role: FOLLOWER
I20240502 14:04:38.036722 12889 consensus_queue.cc:260] T dec6810c74994cb2b031154d236a6047 P 86c24be0852c4662926a2951c7e19394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c24be0852c4662926a2951c7e19394" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 46227 } }
I20240502 14:04:38.037395 12889 raft_consensus.cc:388] T dec6810c74994cb2b031154d236a6047 P 86c24be0852c4662926a2951c7e19394 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:04:38.037755 12889 raft_consensus.cc:482] T dec6810c74994cb2b031154d236a6047 P 86c24be0852c4662926a2951c7e19394 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:04:38.038053 12889 raft_consensus.cc:3037] T dec6810c74994cb2b031154d236a6047 P 86c24be0852c4662926a2951c7e19394 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:38.038913 12890 raft_consensus.cc:348] T b8d9fc8406104e59b6677c5eb048d950 P ec0283e4f52e4572a67722ccc1097dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0283e4f52e4572a67722ccc1097dad" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46749 } }
I20240502 14:04:38.039449 12890 raft_consensus.cc:374] T b8d9fc8406104e59b6677c5eb048d950 P ec0283e4f52e4572a67722ccc1097dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:38.039717 12890 raft_consensus.cc:729] T b8d9fc8406104e59b6677c5eb048d950 P ec0283e4f52e4572a67722ccc1097dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec0283e4f52e4572a67722ccc1097dad, State: Initialized, Role: FOLLOWER
I20240502 14:04:38.040411 12890 consensus_queue.cc:260] T b8d9fc8406104e59b6677c5eb048d950 P ec0283e4f52e4572a67722ccc1097dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0283e4f52e4572a67722ccc1097dad" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46749 } }
I20240502 14:04:38.041025 12890 raft_consensus.cc:388] T b8d9fc8406104e59b6677c5eb048d950 P ec0283e4f52e4572a67722ccc1097dad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:04:38.041323 12890 raft_consensus.cc:482] T b8d9fc8406104e59b6677c5eb048d950 P ec0283e4f52e4572a67722ccc1097dad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:04:38.041704 12890 raft_consensus.cc:3037] T b8d9fc8406104e59b6677c5eb048d950 P ec0283e4f52e4572a67722ccc1097dad [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:38.042541 12888 raft_consensus.cc:504] T 571e065c13df4042a262adca43c13541 P 688d5aecbeb74735aecb64a747dee84e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688d5aecbeb74735aecb64a747dee84e" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35101 } }
I20240502 14:04:38.043246 12888 leader_election.cc:302] T 571e065c13df4042a262adca43c13541 P 688d5aecbeb74735aecb64a747dee84e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 688d5aecbeb74735aecb64a747dee84e; no voters:
I20240502 14:04:38.044346 12889 raft_consensus.cc:504] T dec6810c74994cb2b031154d236a6047 P 86c24be0852c4662926a2951c7e19394 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c24be0852c4662926a2951c7e19394" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 46227 } }
I20240502 14:04:38.044780 12888 leader_election.cc:288] T 571e065c13df4042a262adca43c13541 P 688d5aecbeb74735aecb64a747dee84e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:04:38.045107 12889 leader_election.cc:302] T dec6810c74994cb2b031154d236a6047 P 86c24be0852c4662926a2951c7e19394 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86c24be0852c4662926a2951c7e19394; no voters:
I20240502 14:04:38.045236 12894 raft_consensus.cc:2781] T 571e065c13df4042a262adca43c13541 P 688d5aecbeb74735aecb64a747dee84e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:38.049069 12890 raft_consensus.cc:504] T b8d9fc8406104e59b6677c5eb048d950 P ec0283e4f52e4572a67722ccc1097dad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0283e4f52e4572a67722ccc1097dad" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46749 } }
I20240502 14:04:38.050015 12890 leader_election.cc:302] T b8d9fc8406104e59b6677c5eb048d950 P ec0283e4f52e4572a67722ccc1097dad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec0283e4f52e4572a67722ccc1097dad; no voters:
I20240502 14:04:38.052357 12890 leader_election.cc:288] T b8d9fc8406104e59b6677c5eb048d950 P ec0283e4f52e4572a67722ccc1097dad [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:04:38.052798 12898 raft_consensus.cc:2781] T b8d9fc8406104e59b6677c5eb048d950 P ec0283e4f52e4572a67722ccc1097dad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:38.054041 12889 leader_election.cc:288] T dec6810c74994cb2b031154d236a6047 P 86c24be0852c4662926a2951c7e19394 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:04:38.054343 12896 raft_consensus.cc:2781] T dec6810c74994cb2b031154d236a6047 P 86c24be0852c4662926a2951c7e19394 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:38.055028 12888 ts_tablet_manager.cc:1398] T 571e065c13df4042a262adca43c13541 P 688d5aecbeb74735aecb64a747dee84e: Time spent starting tablet: real 0.026s user 0.016s sys 0.009s
I20240502 14:04:38.055804 12894 raft_consensus.cc:686] T 571e065c13df4042a262adca43c13541 P 688d5aecbeb74735aecb64a747dee84e [term 1 LEADER]: Becoming Leader. State: Replica: 688d5aecbeb74735aecb64a747dee84e, State: Running, Role: LEADER
I20240502 14:04:38.056838 12894 consensus_queue.cc:237] T 571e065c13df4042a262adca43c13541 P 688d5aecbeb74735aecb64a747dee84e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688d5aecbeb74735aecb64a747dee84e" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35101 } }
I20240502 14:04:38.059938 12890 ts_tablet_manager.cc:1398] T b8d9fc8406104e59b6677c5eb048d950 P ec0283e4f52e4572a67722ccc1097dad: Time spent starting tablet: real 0.024s user 0.017s sys 0.001s
I20240502 14:04:38.060801 12896 raft_consensus.cc:686] T dec6810c74994cb2b031154d236a6047 P 86c24be0852c4662926a2951c7e19394 [term 1 LEADER]: Becoming Leader. State: Replica: 86c24be0852c4662926a2951c7e19394, State: Running, Role: LEADER
I20240502 14:04:38.061841 12898 raft_consensus.cc:686] T b8d9fc8406104e59b6677c5eb048d950 P ec0283e4f52e4572a67722ccc1097dad [term 1 LEADER]: Becoming Leader. State: Replica: ec0283e4f52e4572a67722ccc1097dad, State: Running, Role: LEADER
I20240502 14:04:38.061630 12896 consensus_queue.cc:237] T dec6810c74994cb2b031154d236a6047 P 86c24be0852c4662926a2951c7e19394 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c24be0852c4662926a2951c7e19394" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 46227 } }
I20240502 14:04:38.062530 12898 consensus_queue.cc:237] T b8d9fc8406104e59b6677c5eb048d950 P ec0283e4f52e4572a67722ccc1097dad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0283e4f52e4572a67722ccc1097dad" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46749 } }
I20240502 14:04:38.065501 12889 ts_tablet_manager.cc:1398] T dec6810c74994cb2b031154d236a6047 P 86c24be0852c4662926a2951c7e19394: Time spent starting tablet: real 0.032s user 0.015s sys 0.017s
I20240502 14:04:38.071285 12534 catalog_manager.cc:5484] T 571e065c13df4042a262adca43c13541 P 688d5aecbeb74735aecb64a747dee84e reported cstate change: term changed from 0 to 1, leader changed from <none> to 688d5aecbeb74735aecb64a747dee84e (127.11.9.67). New cstate: current_term: 1 leader_uuid: "688d5aecbeb74735aecb64a747dee84e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688d5aecbeb74735aecb64a747dee84e" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35101 } health_report { overall_health: HEALTHY } } }
I20240502 14:04:38.079514 12533 catalog_manager.cc:5484] T dec6810c74994cb2b031154d236a6047 P 86c24be0852c4662926a2951c7e19394 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86c24be0852c4662926a2951c7e19394 (127.11.9.65). New cstate: current_term: 1 leader_uuid: "86c24be0852c4662926a2951c7e19394" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c24be0852c4662926a2951c7e19394" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 46227 } health_report { overall_health: HEALTHY } } }
I20240502 14:04:38.093760 12532 catalog_manager.cc:5484] T b8d9fc8406104e59b6677c5eb048d950 P ec0283e4f52e4572a67722ccc1097dad reported cstate change: term changed from 0 to 1, leader changed from <none> to ec0283e4f52e4572a67722ccc1097dad (127.11.9.66). New cstate: current_term: 1 leader_uuid: "ec0283e4f52e4572a67722ccc1097dad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0283e4f52e4572a67722ccc1097dad" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46749 } health_report { overall_health: HEALTHY } } }
I20240502 14:04:38.176214 11301 master.cc:537] Master@127.11.9.124:45685 shutting down...
I20240502 14:04:38.190801 11301 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:04:38.191519 11301 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:38.191807 11301 tablet_replica.cc:330] T 00000000000000000000000000000000 P 91973bdfb46543e9b95f811246c6377f: stopping tablet replica
I20240502 14:04:38.211555 11301 master.cc:552] Master@127.11.9.124:45685 shutdown complete.
W20240502 14:04:39.116732 12852 heartbeater.cc:644] Failed to heartbeat to 127.11.9.124:45685 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.9.124:45685: connect: Connection refused (error 111)
I20240502 14:04:39.654314 12914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 91973bdfb46543e9b95f811246c6377f)
I20240502 14:04:39.654789 12914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } }
I20240502 14:04:39.655544 12915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 91973bdfb46543e9b95f811246c6377f)
I20240502 14:04:39.655941 12915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } }
I20240502 14:04:39.656666 12914 leader_election.cc:288] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d445a06bca1419eaab962838f24c742 (127.11.9.125:40629), 91973bdfb46543e9b95f811246c6377f (127.11.9.124:45685)
I20240502 14:04:39.657438 12478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "3d445a06bca1419eaab962838f24c742" is_pre_election: true
I20240502 14:04:39.658190 12478 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e1c6a2504b84982b5bb3cf32bff6e48 in term 1.
I20240502 14:04:39.658346 12413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d445a06bca1419eaab962838f24c742" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" is_pre_election: true
I20240502 14:04:39.658948 12413 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d445a06bca1419eaab962838f24c742 in term 1.
I20240502 14:04:39.660382 12387 leader_election.cc:302] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1c6a2504b84982b5bb3cf32bff6e48, 3d445a06bca1419eaab962838f24c742; no voters:
I20240502 14:04:39.660637 12453 leader_election.cc:302] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1c6a2504b84982b5bb3cf32bff6e48, 3d445a06bca1419eaab962838f24c742; no voters:
I20240502 14:04:39.661238 12914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:04:39.661415 12915 leader_election.cc:288] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e1c6a2504b84982b5bb3cf32bff6e48 (127.11.9.126:35153), 91973bdfb46543e9b95f811246c6377f (127.11.9.124:45685)
I20240502 14:04:39.661753 12914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 91973bdfb46543e9b95f811246c6377f)
I20240502 14:04:39.662155 12915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:04:39.662322 12914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:04:39.662688 12915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 91973bdfb46543e9b95f811246c6377f)
I20240502 14:04:39.663206 12915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:04:39.668272 12388 leader_election.cc:334] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 91973bdfb46543e9b95f811246c6377f (127.11.9.124:45685): Network error: Client connection negotiation failed: client connection to 127.11.9.124:45685: connect: Connection refused (error 111)
I20240502 14:04:39.669991 12914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } }
W20240502 14:04:39.671999 12453 leader_election.cc:334] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 91973bdfb46543e9b95f811246c6377f (127.11.9.124:45685): Network error: Client connection negotiation failed: client connection to 127.11.9.124:45685: connect: Connection refused (error 111)
I20240502 14:04:39.672523 12478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "3d445a06bca1419eaab962838f24c742"
I20240502 14:04:39.672380 12915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } }
I20240502 14:04:39.672906 12914 leader_election.cc:288] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [CANDIDATE]: Term 2 election: Requested vote from peers 3d445a06bca1419eaab962838f24c742 (127.11.9.125:40629), 91973bdfb46543e9b95f811246c6377f (127.11.9.124:45685)
I20240502 14:04:39.673444 12478 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0e1c6a2504b84982b5bb3cf32bff6e48 in current term 2: Already voted for candidate 3d445a06bca1419eaab962838f24c742 in this term.
I20240502 14:04:39.675000 12915 leader_election.cc:288] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [CANDIDATE]: Term 2 election: Requested vote from peers 0e1c6a2504b84982b5bb3cf32bff6e48 (127.11.9.126:35153), 91973bdfb46543e9b95f811246c6377f (127.11.9.124:45685)
W20240502 14:04:39.675704 12388 leader_election.cc:334] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 91973bdfb46543e9b95f811246c6377f (127.11.9.124:45685): Network error: Client connection negotiation failed: client connection to 127.11.9.124:45685: connect: Connection refused (error 111)
I20240502 14:04:39.675772 12413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d445a06bca1419eaab962838f24c742" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48"
I20240502 14:04:39.676272 12388 leader_election.cc:302] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e1c6a2504b84982b5bb3cf32bff6e48; no voters: 3d445a06bca1419eaab962838f24c742, 91973bdfb46543e9b95f811246c6377f
I20240502 14:04:39.676605 12413 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d445a06bca1419eaab962838f24c742 in current term 2: Already voted for candidate 0e1c6a2504b84982b5bb3cf32bff6e48 in this term.
I20240502 14:04:39.677024 12914 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240502 14:04:39.678483 12453 leader_election.cc:334] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 91973bdfb46543e9b95f811246c6377f (127.11.9.124:45685): Network error: Client connection negotiation failed: client connection to 127.11.9.124:45685: connect: Connection refused (error 111)
I20240502 14:04:39.678766 12453 leader_election.cc:302] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d445a06bca1419eaab962838f24c742; no voters: 0e1c6a2504b84982b5bb3cf32bff6e48, 91973bdfb46543e9b95f811246c6377f
I20240502 14:04:39.679231 12915 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 14:04:41.450241 12928 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:41.450922 12928 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } }
I20240502 14:04:41.452632 12928 leader_election.cc:288] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e1c6a2504b84982b5bb3cf32bff6e48 (127.11.9.126:35153), 91973bdfb46543e9b95f811246c6377f (127.11.9.124:45685)
I20240502 14:04:41.453925 12413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d445a06bca1419eaab962838f24c742" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" is_pre_election: true
I20240502 14:04:41.454788 12413 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d445a06bca1419eaab962838f24c742 in term 2.
W20240502 14:04:41.455549 12453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.9.124:45685: connect: Connection refused (error 111) [suppressed 17 similar messages]
I20240502 14:04:41.458354 12453 leader_election.cc:302] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1c6a2504b84982b5bb3cf32bff6e48, 3d445a06bca1419eaab962838f24c742; no voters:
I20240502 14:04:41.458954 12928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 14:04:41.459267 12928 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:41.459529 12928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [term 2 FOLLOWER]: Advancing to term 3
W20240502 14:04:41.460232 12453 leader_election.cc:334] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 91973bdfb46543e9b95f811246c6377f (127.11.9.124:45685): Network error: Client connection negotiation failed: client connection to 127.11.9.124:45685: connect: Connection refused (error 111)
I20240502 14:04:41.465014 12928 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } }
I20240502 14:04:41.466434 12928 leader_election.cc:288] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [CANDIDATE]: Term 3 election: Requested vote from peers 0e1c6a2504b84982b5bb3cf32bff6e48 (127.11.9.126:35153), 91973bdfb46543e9b95f811246c6377f (127.11.9.124:45685)
I20240502 14:04:41.467105 12413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d445a06bca1419eaab962838f24c742" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48"
I20240502 14:04:41.467558 12413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [term 2 FOLLOWER]: Advancing to term 3
W20240502 14:04:41.469362 12453 leader_election.cc:334] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 91973bdfb46543e9b95f811246c6377f (127.11.9.124:45685): Network error: Client connection negotiation failed: client connection to 127.11.9.124:45685: connect: Connection refused (error 111)
I20240502 14:04:41.472400 12413 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d445a06bca1419eaab962838f24c742 in term 3.
I20240502 14:04:41.473176 12453 leader_election.cc:302] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0e1c6a2504b84982b5bb3cf32bff6e48, 3d445a06bca1419eaab962838f24c742; no voters: 91973bdfb46543e9b95f811246c6377f
I20240502 14:04:41.473798 12928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:04:41.474732 12928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [term 3 LEADER]: Becoming Leader. State: Replica: 3d445a06bca1419eaab962838f24c742, State: Running, Role: LEADER
I20240502 14:04:41.475492 12928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } }
I20240502 14:04:41.479269 12931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d445a06bca1419eaab962838f24c742. Latest consensus state: current_term: 3 leader_uuid: "3d445a06bca1419eaab962838f24c742" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } } }
I20240502 14:04:41.480123 12931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [sys.catalog]: This master's current role is: LEADER
I20240502 14:04:41.481539 12933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:04:41.486964 12933 catalog_manager.cc:669] Loaded metadata for table test-workload [id=735f9b44aa674bfdbe05ad3cc83f3a46]
I20240502 14:04:41.491364 12933 tablet_loader.cc:96] loaded metadata for tablet 571e065c13df4042a262adca43c13541 (table test-workload [id=735f9b44aa674bfdbe05ad3cc83f3a46])
I20240502 14:04:41.493693 12933 tablet_loader.cc:96] loaded metadata for tablet b8d9fc8406104e59b6677c5eb048d950 (table test-workload [id=735f9b44aa674bfdbe05ad3cc83f3a46])
I20240502 14:04:41.494508 12933 tablet_loader.cc:96] loaded metadata for tablet dec6810c74994cb2b031154d236a6047 (table test-workload [id=735f9b44aa674bfdbe05ad3cc83f3a46])
I20240502 14:04:41.495251 12933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:04:41.498062 12933 catalog_manager.cc:1259] Loaded cluster ID: d5cabb8e01cc4ca98e4bac6421bea4be
I20240502 14:04:41.498252 12933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:04:41.501538 12933 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:04:41.504127 12933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742: Loaded TSK: 0
I20240502 14:04:41.504954 12933 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:04:41.825616 12586 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:04:41.825925 12586 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:04:41.854610 11301 tablet_server.cc:178] TabletServer@127.11.9.65:0 shutting down...
I20240502 14:04:41.875928 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:04:41.876596 11301 tablet_replica.cc:330] T dec6810c74994cb2b031154d236a6047 P 86c24be0852c4662926a2951c7e19394: stopping tablet replica
I20240502 14:04:41.877189 11301 raft_consensus.cc:2229] T dec6810c74994cb2b031154d236a6047 P 86c24be0852c4662926a2951c7e19394 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:04:41.877727 11301 raft_consensus.cc:2258] T dec6810c74994cb2b031154d236a6047 P 86c24be0852c4662926a2951c7e19394 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:41.896129 11301 tablet_server.cc:194] TabletServer@127.11.9.65:0 shutdown complete.
I20240502 14:04:41.906101 11301 tablet_server.cc:178] TabletServer@127.11.9.66:0 shutting down...
W20240502 14:04:41.920858 12453 consensus_peers.cc:485] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 -> Peer 91973bdfb46543e9b95f811246c6377f (127.11.9.124:45685): Couldn't send request to peer 91973bdfb46543e9b95f811246c6377f. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.124:45685: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:04:41.931376 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:04:41.932044 11301 tablet_replica.cc:330] T b8d9fc8406104e59b6677c5eb048d950 P ec0283e4f52e4572a67722ccc1097dad: stopping tablet replica
I20240502 14:04:41.932569 11301 raft_consensus.cc:2229] T b8d9fc8406104e59b6677c5eb048d950 P ec0283e4f52e4572a67722ccc1097dad [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:04:41.933048 11301 raft_consensus.cc:2258] T b8d9fc8406104e59b6677c5eb048d950 P ec0283e4f52e4572a67722ccc1097dad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:41.951511 11301 tablet_server.cc:194] TabletServer@127.11.9.66:0 shutdown complete.
I20240502 14:04:41.956936 12413 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [term 3 FOLLOWER]: Refusing update from remote peer 3d445a06bca1419eaab962838f24c742: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240502 14:04:41.958437 12931 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20240502 14:04:41.962879 11301 tablet_server.cc:178] TabletServer@127.11.9.67:0 shutting down...
I20240502 14:04:41.964754 12934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d445a06bca1419eaab962838f24c742. Latest consensus state: current_term: 3 leader_uuid: "3d445a06bca1419eaab962838f24c742" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } } }
I20240502 14:04:41.966820 12934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:04:41.977340 12931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "3d445a06bca1419eaab962838f24c742" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } } }
I20240502 14:04:41.978240 12931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [sys.catalog]: This master's current role is: LEADER
I20240502 14:04:41.979055 12934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "3d445a06bca1419eaab962838f24c742" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1c6a2504b84982b5bb3cf32bff6e48" member_type: VOTER last_known_addr { host: "127.11.9.126" port: 35153 } } peers { permanent_uuid: "3d445a06bca1419eaab962838f24c742" member_type: VOTER last_known_addr { host: "127.11.9.125" port: 40629 } } peers { permanent_uuid: "91973bdfb46543e9b95f811246c6377f" member_type: VOTER last_known_addr { host: "127.11.9.124" port: 45685 } } }
I20240502 14:04:41.980018 12934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 14:04:41.995220 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:04:41.995872 11301 tablet_replica.cc:330] T 571e065c13df4042a262adca43c13541 P 688d5aecbeb74735aecb64a747dee84e: stopping tablet replica
I20240502 14:04:41.996393 11301 raft_consensus.cc:2229] T 571e065c13df4042a262adca43c13541 P 688d5aecbeb74735aecb64a747dee84e [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:04:41.996907 11301 raft_consensus.cc:2258] T 571e065c13df4042a262adca43c13541 P 688d5aecbeb74735aecb64a747dee84e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:42.015151 11301 tablet_server.cc:194] TabletServer@127.11.9.67:0 shutdown complete.
I20240502 14:04:42.024482 11301 master.cc:537] Master@127.11.9.126:35153 shutting down...
I20240502 14:04:42.041662 11301 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:42.042333 11301 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:42.042666 11301 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0e1c6a2504b84982b5bb3cf32bff6e48: stopping tablet replica
I20240502 14:04:42.060899 11301 master.cc:552] Master@127.11.9.126:35153 shutdown complete.
I20240502 14:04:42.073964 11301 master.cc:537] Master@127.11.9.125:40629 shutting down...
I20240502 14:04:42.090592 11301 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [term 3 LEADER]: Raft consensus shutting down.
I20240502 14:04:42.091440 11301 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:42.091833 11301 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3d445a06bca1419eaab962838f24c742: stopping tablet replica
I20240502 14:04:42.110209 11301 master.cc:552] Master@127.11.9.125:40629 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (6656 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240502 14:04:42.140237 11301 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.9.126:40193
I20240502 14:04:42.141268 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:42.147375 12938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:42.148160 12939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:42.148861 12941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:42.149683 11301 server_base.cc:1034] running on GCE node
I20240502 14:04:42.150449 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:04:42.150629 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:04:42.150785 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658682150763 us; error 0 us; skew 500 ppm
I20240502 14:04:42.151243 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:04:42.157298 11301 webserver.cc:460] Webserver started at http://127.11.9.126:34449/ using document root <none> and password file <none>
I20240502 14:04:42.157869 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:42.158036 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:42.158290 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:42.159457 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658659912502-11301-0/minicluster-data/master-0-root/instance:
uuid: "5c48abe57dc8447d8b6d4bed9d17be45"
format_stamp: "Formatted at 2024-05-02 14:04:42 on dist-test-slave-j72s"
I20240502 14:04:42.163741 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 14:04:42.166714 12946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:42.167385 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:04:42.167631 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658659912502-11301-0/minicluster-data/master-0-root
uuid: "5c48abe57dc8447d8b6d4bed9d17be45"
format_stamp: "Formatted at 2024-05-02 14:04:42 on dist-test-slave-j72s"
I20240502 14:04:42.167870 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658659912502-11301-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658659912502-11301-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658659912502-11301-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:42.207242 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:42.208328 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:42.242799 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.126:40193
I20240502 14:04:42.242888 12997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.126:40193 every 8 connection(s)
I20240502 14:04:42.246301 12998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:42.255244 12998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c48abe57dc8447d8b6d4bed9d17be45: Bootstrap starting.
I20240502 14:04:42.259099 12998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c48abe57dc8447d8b6d4bed9d17be45: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:42.262785 12998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c48abe57dc8447d8b6d4bed9d17be45: No bootstrap required, opened a new log
I20240502 14:04:42.264473 12998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c48abe57dc8447d8b6d4bed9d17be45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c48abe57dc8447d8b6d4bed9d17be45" member_type: VOTER }
I20240502 14:04:42.264811 12998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c48abe57dc8447d8b6d4bed9d17be45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:42.264979 12998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c48abe57dc8447d8b6d4bed9d17be45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c48abe57dc8447d8b6d4bed9d17be45, State: Initialized, Role: FOLLOWER
I20240502 14:04:42.265457 12998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c48abe57dc8447d8b6d4bed9d17be45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c48abe57dc8447d8b6d4bed9d17be45" member_type: VOTER }
I20240502 14:04:42.265931 12998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c48abe57dc8447d8b6d4bed9d17be45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:04:42.266119 12998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c48abe57dc8447d8b6d4bed9d17be45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:04:42.266314 12998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c48abe57dc8447d8b6d4bed9d17be45 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:42.270210 12998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c48abe57dc8447d8b6d4bed9d17be45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c48abe57dc8447d8b6d4bed9d17be45" member_type: VOTER }
I20240502 14:04:42.270627 12998 leader_election.cc:302] T 00000000000000000000000000000000 P 5c48abe57dc8447d8b6d4bed9d17be45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c48abe57dc8447d8b6d4bed9d17be45; no voters:
I20240502 14:04:42.271628 12998 leader_election.cc:288] T 00000000000000000000000000000000 P 5c48abe57dc8447d8b6d4bed9d17be45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:04:42.271890 13001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c48abe57dc8447d8b6d4bed9d17be45 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:42.273212 13001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c48abe57dc8447d8b6d4bed9d17be45 [term 1 LEADER]: Becoming Leader. State: Replica: 5c48abe57dc8447d8b6d4bed9d17be45, State: Running, Role: LEADER
I20240502 14:04:42.273897 13001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c48abe57dc8447d8b6d4bed9d17be45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c48abe57dc8447d8b6d4bed9d17be45" member_type: VOTER }
I20240502 14:04:42.274498 12998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c48abe57dc8447d8b6d4bed9d17be45 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:04:42.276522 13003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c48abe57dc8447d8b6d4bed9d17be45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c48abe57dc8447d8b6d4bed9d17be45. Latest consensus state: current_term: 1 leader_uuid: "5c48abe57dc8447d8b6d4bed9d17be45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c48abe57dc8447d8b6d4bed9d17be45" member_type: VOTER } }
I20240502 14:04:42.277113 13003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c48abe57dc8447d8b6d4bed9d17be45 [sys.catalog]: This master's current role is: LEADER
I20240502 14:04:42.277972 13002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c48abe57dc8447d8b6d4bed9d17be45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c48abe57dc8447d8b6d4bed9d17be45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c48abe57dc8447d8b6d4bed9d17be45" member_type: VOTER } }
I20240502 14:04:42.278728 13002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c48abe57dc8447d8b6d4bed9d17be45 [sys.catalog]: This master's current role is: LEADER
I20240502 14:04:42.281219 13007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:04:42.285503 13007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:04:42.290365 11301 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:04:42.294185 13007 catalog_manager.cc:1347] Generated new cluster ID: 4b17bda3f5664a348981823980d2c78e
I20240502 14:04:42.294461 13007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:04:42.307619 13007 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:04:42.308748 13007 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:04:42.326197 13007 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5c48abe57dc8447d8b6d4bed9d17be45: Generated new TSK 0
I20240502 14:04:42.326761 13007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:04:42.357803 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:42.364120 13019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:42.365177 13020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:42.366118 13022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:42.367200 11301 server_base.cc:1034] running on GCE node
I20240502 14:04:42.367848 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:04:42.368014 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:04:42.368161 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658682368142 us; error 0 us; skew 500 ppm
I20240502 14:04:42.368670 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:04:42.370837 11301 webserver.cc:460] Webserver started at http://127.11.9.65:38649/ using document root <none> and password file <none>
I20240502 14:04:42.371270 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:42.371434 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:42.371681 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:42.372771 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658659912502-11301-0/minicluster-data/ts-0-root/instance:
uuid: "8f41e9045e714e50af94553f17bb0ae3"
format_stamp: "Formatted at 2024-05-02 14:04:42 on dist-test-slave-j72s"
I20240502 14:04:42.377084 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20240502 14:04:42.379992 13027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:42.380676 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:04:42.380921 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658659912502-11301-0/minicluster-data/ts-0-root
uuid: "8f41e9045e714e50af94553f17bb0ae3"
format_stamp: "Formatted at 2024-05-02 14:04:42 on dist-test-slave-j72s"
I20240502 14:04:42.381189 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658659912502-11301-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658659912502-11301-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658659912502-11301-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:42.407718 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:42.408821 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:42.410250 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:04:42.412431 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:42.412614 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:42.412832 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:42.412993 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:42.450712 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.65:41711
I20240502 14:04:42.450794 13089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.65:41711 every 8 connection(s)
I20240502 14:04:42.455076 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:42.463502 13095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:42.464380 13094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:42.466261 13097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:42.467056 11301 server_base.cc:1034] running on GCE node
I20240502 14:04:42.467808 13090 heartbeater.cc:347] Connected to a master server at 127.11.9.126:40193
I20240502 14:04:42.467936 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 14:04:42.468140 13090 heartbeater.cc:464] Registering TS with master...
W20240502 14:04:42.468220 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:04:42.468518 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658682468502 us; error 0 us; skew 500 ppm
I20240502 14:04:42.469089 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:04:42.469041 13090 heartbeater.cc:511] Master 127.11.9.126:40193 requested a full tablet report, sending...
I20240502 14:04:42.471385 12963 ts_manager.cc:194] Registered new tserver with Master: 8f41e9045e714e50af94553f17bb0ae3 (127.11.9.65:41711)
I20240502 14:04:42.471908 11301 webserver.cc:460] Webserver started at http://127.11.9.66:38839/ using document root <none> and password file <none>
I20240502 14:04:42.472555 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:42.472813 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:42.473137 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:42.473819 12963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50356
I20240502 14:04:42.474481 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658659912502-11301-0/minicluster-data/ts-1-root/instance:
uuid: "1211bad1f809400e953f4d1642886b3d"
format_stamp: "Formatted at 2024-05-02 14:04:42 on dist-test-slave-j72s"
I20240502 14:04:42.478720 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240502 14:04:42.481585 13102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:42.482267 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:04:42.482540 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658659912502-11301-0/minicluster-data/ts-1-root
uuid: "1211bad1f809400e953f4d1642886b3d"
format_stamp: "Formatted at 2024-05-02 14:04:42 on dist-test-slave-j72s"
I20240502 14:04:42.482841 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658659912502-11301-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658659912502-11301-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658659912502-11301-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:42.504477 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:42.505465 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:42.506830 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:04:42.508932 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:42.509105 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:42.509303 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:42.509439 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:42.544749 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.66:38837
I20240502 14:04:42.544844 13164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.66:38837 every 8 connection(s)
I20240502 14:04:42.549158 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:42.556116 13168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:42.556874 13169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:42.559729 13171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:42.560390 11301 server_base.cc:1034] running on GCE node
I20240502 14:04:42.560910 13165 heartbeater.cc:347] Connected to a master server at 127.11.9.126:40193
I20240502 14:04:42.561323 13165 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:42.561369 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:04:42.561751 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:04:42.561969 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658682561951 us; error 0 us; skew 500 ppm
I20240502 14:04:42.562343 13165 heartbeater.cc:511] Master 127.11.9.126:40193 requested a full tablet report, sending...
I20240502 14:04:42.562743 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:04:42.564577 12963 ts_manager.cc:194] Registered new tserver with Master: 1211bad1f809400e953f4d1642886b3d (127.11.9.66:38837)
I20240502 14:04:42.565402 11301 webserver.cc:460] Webserver started at http://127.11.9.67:34265/ using document root <none> and password file <none>
I20240502 14:04:42.565951 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:42.566131 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:42.566051 12963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50366
I20240502 14:04:42.566488 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:42.567951 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658659912502-11301-0/minicluster-data/ts-2-root/instance:
uuid: "c14a1e9fa43944fb952510b90e79bb30"
format_stamp: "Formatted at 2024-05-02 14:04:42 on dist-test-slave-j72s"
I20240502 14:04:42.572057 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240502 14:04:42.575192 13176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:42.575935 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:04:42.576200 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658659912502-11301-0/minicluster-data/ts-2-root
uuid: "c14a1e9fa43944fb952510b90e79bb30"
format_stamp: "Formatted at 2024-05-02 14:04:42 on dist-test-slave-j72s"
I20240502 14:04:42.576436 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658659912502-11301-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658659912502-11301-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658659912502-11301-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:42.594287 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:42.595285 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:42.596601 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:04:42.598779 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:42.598985 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:42.599213 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:42.599352 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:42.637506 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.67:34847
I20240502 14:04:42.637620 13238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.67:34847 every 8 connection(s)
I20240502 14:04:42.649312 13239 heartbeater.cc:347] Connected to a master server at 127.11.9.126:40193
I20240502 14:04:42.649652 13239 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:42.650274 13239 heartbeater.cc:511] Master 127.11.9.126:40193 requested a full tablet report, sending...
I20240502 14:04:42.651971 12963 ts_manager.cc:194] Registered new tserver with Master: c14a1e9fa43944fb952510b90e79bb30 (127.11.9.67:34847)
I20240502 14:04:42.652436 11301 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012019667s
I20240502 14:04:42.653224 12963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50382
I20240502 14:04:43.476248 13090 heartbeater.cc:503] Master 127.11.9.126:40193 was elected leader, sending a full tablet report...
I20240502 14:04:43.568585 13165 heartbeater.cc:503] Master 127.11.9.126:40193 was elected leader, sending a full tablet report...
I20240502 14:04:43.655198 13239 heartbeater.cc:503] Master 127.11.9.126:40193 was elected leader, sending a full tablet report...
I20240502 14:04:43.683619 11301 test_util.cc:268] Using random seed: 364937997
I20240502 14:04:43.703971 12963 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50390:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:04:43.705969 12963 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:04:43.748566 13130 tablet_service.cc:1467] Processing CreateTablet for tablet 4b48c7659b094e068c680f9b27fbcdee (DEFAULT_TABLE table=test-workload [id=fd4179708e91461e8b7e0832a19c11ac]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 14:04:43.748769 13129 tablet_service.cc:1467] Processing CreateTablet for tablet 192ef8db613a46d98e647871c8180118 (DEFAULT_TABLE table=test-workload [id=fd4179708e91461e8b7e0832a19c11ac]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 14:04:43.750041 13130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b48c7659b094e068c680f9b27fbcdee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:43.750614 13129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192ef8db613a46d98e647871c8180118. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:43.752122 13055 tablet_service.cc:1467] Processing CreateTablet for tablet 4b48c7659b094e068c680f9b27fbcdee (DEFAULT_TABLE table=test-workload [id=fd4179708e91461e8b7e0832a19c11ac]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 14:04:43.753401 13055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b48c7659b094e068c680f9b27fbcdee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:43.754462 13054 tablet_service.cc:1467] Processing CreateTablet for tablet 192ef8db613a46d98e647871c8180118 (DEFAULT_TABLE table=test-workload [id=fd4179708e91461e8b7e0832a19c11ac]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 14:04:43.755620 13054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192ef8db613a46d98e647871c8180118. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:43.759151 13204 tablet_service.cc:1467] Processing CreateTablet for tablet 4b48c7659b094e068c680f9b27fbcdee (DEFAULT_TABLE table=test-workload [id=fd4179708e91461e8b7e0832a19c11ac]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240502 14:04:43.760370 13204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b48c7659b094e068c680f9b27fbcdee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:43.767043 13203 tablet_service.cc:1467] Processing CreateTablet for tablet 192ef8db613a46d98e647871c8180118 (DEFAULT_TABLE table=test-workload [id=fd4179708e91461e8b7e0832a19c11ac]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240502 14:04:43.768219 13203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192ef8db613a46d98e647871c8180118. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:43.783169 13260 tablet_bootstrap.cc:492] T 192ef8db613a46d98e647871c8180118 P 8f41e9045e714e50af94553f17bb0ae3: Bootstrap starting.
I20240502 14:04:43.791676 13260 tablet_bootstrap.cc:654] T 192ef8db613a46d98e647871c8180118 P 8f41e9045e714e50af94553f17bb0ae3: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:43.794836 13261 tablet_bootstrap.cc:492] T 4b48c7659b094e068c680f9b27fbcdee P c14a1e9fa43944fb952510b90e79bb30: Bootstrap starting.
I20240502 14:04:43.795457 13259 tablet_bootstrap.cc:492] T 192ef8db613a46d98e647871c8180118 P 1211bad1f809400e953f4d1642886b3d: Bootstrap starting.
I20240502 14:04:43.798323 13260 tablet_bootstrap.cc:492] T 192ef8db613a46d98e647871c8180118 P 8f41e9045e714e50af94553f17bb0ae3: No bootstrap required, opened a new log
I20240502 14:04:43.798804 13260 ts_tablet_manager.cc:1367] T 192ef8db613a46d98e647871c8180118 P 8f41e9045e714e50af94553f17bb0ae3: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.000s
I20240502 14:04:43.800365 13259 tablet_bootstrap.cc:654] T 192ef8db613a46d98e647871c8180118 P 1211bad1f809400e953f4d1642886b3d: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:43.800956 13261 tablet_bootstrap.cc:654] T 4b48c7659b094e068c680f9b27fbcdee P c14a1e9fa43944fb952510b90e79bb30: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:43.801398 13260 raft_consensus.cc:348] T 192ef8db613a46d98e647871c8180118 P 8f41e9045e714e50af94553f17bb0ae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } }
I20240502 14:04:43.803361 13260 raft_consensus.cc:374] T 192ef8db613a46d98e647871c8180118 P 8f41e9045e714e50af94553f17bb0ae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:43.803741 13260 raft_consensus.cc:729] T 192ef8db613a46d98e647871c8180118 P 8f41e9045e714e50af94553f17bb0ae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f41e9045e714e50af94553f17bb0ae3, State: Initialized, Role: FOLLOWER
I20240502 14:04:43.805850 13259 tablet_bootstrap.cc:492] T 192ef8db613a46d98e647871c8180118 P 1211bad1f809400e953f4d1642886b3d: No bootstrap required, opened a new log
I20240502 14:04:43.805331 13260 consensus_queue.cc:260] T 192ef8db613a46d98e647871c8180118 P 8f41e9045e714e50af94553f17bb0ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } }
I20240502 14:04:43.806229 13259 ts_tablet_manager.cc:1367] T 192ef8db613a46d98e647871c8180118 P 1211bad1f809400e953f4d1642886b3d: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240502 14:04:43.806236 13261 tablet_bootstrap.cc:492] T 4b48c7659b094e068c680f9b27fbcdee P c14a1e9fa43944fb952510b90e79bb30: No bootstrap required, opened a new log
I20240502 14:04:43.806915 13261 ts_tablet_manager.cc:1367] T 4b48c7659b094e068c680f9b27fbcdee P c14a1e9fa43944fb952510b90e79bb30: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20240502 14:04:43.808305 13260 ts_tablet_manager.cc:1398] T 192ef8db613a46d98e647871c8180118 P 8f41e9045e714e50af94553f17bb0ae3: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240502 14:04:43.809244 13260 tablet_bootstrap.cc:492] T 4b48c7659b094e068c680f9b27fbcdee P 8f41e9045e714e50af94553f17bb0ae3: Bootstrap starting.
I20240502 14:04:43.808877 13259 raft_consensus.cc:348] T 192ef8db613a46d98e647871c8180118 P 1211bad1f809400e953f4d1642886b3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } }
I20240502 14:04:43.809731 13259 raft_consensus.cc:374] T 192ef8db613a46d98e647871c8180118 P 1211bad1f809400e953f4d1642886b3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:43.809609 13261 raft_consensus.cc:348] T 4b48c7659b094e068c680f9b27fbcdee P c14a1e9fa43944fb952510b90e79bb30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } }
I20240502 14:04:43.810117 13259 raft_consensus.cc:729] T 192ef8db613a46d98e647871c8180118 P 1211bad1f809400e953f4d1642886b3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1211bad1f809400e953f4d1642886b3d, State: Initialized, Role: FOLLOWER
I20240502 14:04:43.810423 13261 raft_consensus.cc:374] T 4b48c7659b094e068c680f9b27fbcdee P c14a1e9fa43944fb952510b90e79bb30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:43.810807 13261 raft_consensus.cc:729] T 4b48c7659b094e068c680f9b27fbcdee P c14a1e9fa43944fb952510b90e79bb30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c14a1e9fa43944fb952510b90e79bb30, State: Initialized, Role: FOLLOWER
I20240502 14:04:43.810988 13259 consensus_queue.cc:260] T 192ef8db613a46d98e647871c8180118 P 1211bad1f809400e953f4d1642886b3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } }
I20240502 14:04:43.811506 13261 consensus_queue.cc:260] T 4b48c7659b094e068c680f9b27fbcdee P c14a1e9fa43944fb952510b90e79bb30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } }
I20240502 14:04:43.815609 13260 tablet_bootstrap.cc:654] T 4b48c7659b094e068c680f9b27fbcdee P 8f41e9045e714e50af94553f17bb0ae3: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:43.826587 13261 ts_tablet_manager.cc:1398] T 4b48c7659b094e068c680f9b27fbcdee P c14a1e9fa43944fb952510b90e79bb30: Time spent starting tablet: real 0.019s user 0.006s sys 0.011s
I20240502 14:04:43.827944 13261 tablet_bootstrap.cc:492] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30: Bootstrap starting.
I20240502 14:04:43.829690 13259 ts_tablet_manager.cc:1398] T 192ef8db613a46d98e647871c8180118 P 1211bad1f809400e953f4d1642886b3d: Time spent starting tablet: real 0.023s user 0.016s sys 0.004s
I20240502 14:04:43.835274 13261 tablet_bootstrap.cc:654] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:43.838521 13259 tablet_bootstrap.cc:492] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d: Bootstrap starting.
I20240502 14:04:43.842545 13261 tablet_bootstrap.cc:492] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30: No bootstrap required, opened a new log
I20240502 14:04:43.842986 13261 ts_tablet_manager.cc:1367] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20240502 14:04:43.843757 13259 tablet_bootstrap.cc:654] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d: Neither blocks nor log segments found. Creating new log.
I20240502 14:04:43.843936 13260 tablet_bootstrap.cc:492] T 4b48c7659b094e068c680f9b27fbcdee P 8f41e9045e714e50af94553f17bb0ae3: No bootstrap required, opened a new log
I20240502 14:04:43.844419 13260 ts_tablet_manager.cc:1367] T 4b48c7659b094e068c680f9b27fbcdee P 8f41e9045e714e50af94553f17bb0ae3: Time spent bootstrapping tablet: real 0.035s user 0.015s sys 0.018s
I20240502 14:04:43.845402 13261 raft_consensus.cc:348] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } }
I20240502 14:04:43.846225 13261 raft_consensus.cc:374] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:43.846534 13261 raft_consensus.cc:729] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c14a1e9fa43944fb952510b90e79bb30, State: Initialized, Role: FOLLOWER
I20240502 14:04:43.847019 13260 raft_consensus.cc:348] T 4b48c7659b094e068c680f9b27fbcdee P 8f41e9045e714e50af94553f17bb0ae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } }
I20240502 14:04:43.847290 13261 consensus_queue.cc:260] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } }
I20240502 14:04:43.847756 13260 raft_consensus.cc:374] T 4b48c7659b094e068c680f9b27fbcdee P 8f41e9045e714e50af94553f17bb0ae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:43.847908 13259 tablet_bootstrap.cc:492] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d: No bootstrap required, opened a new log
I20240502 14:04:43.848117 13260 raft_consensus.cc:729] T 4b48c7659b094e068c680f9b27fbcdee P 8f41e9045e714e50af94553f17bb0ae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f41e9045e714e50af94553f17bb0ae3, State: Initialized, Role: FOLLOWER
I20240502 14:04:43.848383 13259 ts_tablet_manager.cc:1367] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20240502 14:04:43.848881 13260 consensus_queue.cc:260] T 4b48c7659b094e068c680f9b27fbcdee P 8f41e9045e714e50af94553f17bb0ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } }
I20240502 14:04:43.849444 13261 ts_tablet_manager.cc:1398] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20240502 14:04:43.850610 13260 ts_tablet_manager.cc:1398] T 4b48c7659b094e068c680f9b27fbcdee P 8f41e9045e714e50af94553f17bb0ae3: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20240502 14:04:43.850780 13259 raft_consensus.cc:348] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } }
I20240502 14:04:43.851451 13259 raft_consensus.cc:374] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:04:43.851725 13259 raft_consensus.cc:729] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1211bad1f809400e953f4d1642886b3d, State: Initialized, Role: FOLLOWER
I20240502 14:04:43.852744 13259 consensus_queue.cc:260] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } }
I20240502 14:04:43.854678 13259 ts_tablet_manager.cc:1398] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 14:04:43.949021 13266 raft_consensus.cc:482] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:43.949445 13266 raft_consensus.cc:504] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } }
I20240502 14:04:43.951412 13266 leader_election.cc:288] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f41e9045e714e50af94553f17bb0ae3 (127.11.9.65:41711), c14a1e9fa43944fb952510b90e79bb30 (127.11.9.67:34847)
I20240502 14:04:43.961927 13065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b48c7659b094e068c680f9b27fbcdee" candidate_uuid: "1211bad1f809400e953f4d1642886b3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f41e9045e714e50af94553f17bb0ae3" is_pre_election: true
I20240502 14:04:43.962671 13065 raft_consensus.cc:2454] T 4b48c7659b094e068c680f9b27fbcdee P 8f41e9045e714e50af94553f17bb0ae3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1211bad1f809400e953f4d1642886b3d in term 0.
I20240502 14:04:43.962970 13214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b48c7659b094e068c680f9b27fbcdee" candidate_uuid: "1211bad1f809400e953f4d1642886b3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c14a1e9fa43944fb952510b90e79bb30" is_pre_election: true
I20240502 14:04:43.963590 13105 leader_election.cc:302] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1211bad1f809400e953f4d1642886b3d, 8f41e9045e714e50af94553f17bb0ae3; no voters:
I20240502 14:04:43.963614 13214 raft_consensus.cc:2454] T 4b48c7659b094e068c680f9b27fbcdee P c14a1e9fa43944fb952510b90e79bb30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1211bad1f809400e953f4d1642886b3d in term 0.
I20240502 14:04:43.964313 13266 raft_consensus.cc:2781] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:43.964602 13266 raft_consensus.cc:482] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:43.964917 13266 raft_consensus.cc:3037] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:43.969851 13266 raft_consensus.cc:504] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } }
I20240502 14:04:43.971220 13266 leader_election.cc:288] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [CANDIDATE]: Term 1 election: Requested vote from peers 8f41e9045e714e50af94553f17bb0ae3 (127.11.9.65:41711), c14a1e9fa43944fb952510b90e79bb30 (127.11.9.67:34847)
I20240502 14:04:43.971788 13065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b48c7659b094e068c680f9b27fbcdee" candidate_uuid: "1211bad1f809400e953f4d1642886b3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f41e9045e714e50af94553f17bb0ae3"
I20240502 14:04:43.972198 13214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b48c7659b094e068c680f9b27fbcdee" candidate_uuid: "1211bad1f809400e953f4d1642886b3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c14a1e9fa43944fb952510b90e79bb30"
I20240502 14:04:43.972390 13065 raft_consensus.cc:3037] T 4b48c7659b094e068c680f9b27fbcdee P 8f41e9045e714e50af94553f17bb0ae3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:43.972771 13214 raft_consensus.cc:3037] T 4b48c7659b094e068c680f9b27fbcdee P c14a1e9fa43944fb952510b90e79bb30 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:43.979310 13214 raft_consensus.cc:2454] T 4b48c7659b094e068c680f9b27fbcdee P c14a1e9fa43944fb952510b90e79bb30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1211bad1f809400e953f4d1642886b3d in term 1.
I20240502 14:04:43.979487 13065 raft_consensus.cc:2454] T 4b48c7659b094e068c680f9b27fbcdee P 8f41e9045e714e50af94553f17bb0ae3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1211bad1f809400e953f4d1642886b3d in term 1.
I20240502 14:04:43.980324 13105 leader_election.cc:302] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1211bad1f809400e953f4d1642886b3d, c14a1e9fa43944fb952510b90e79bb30; no voters:
I20240502 14:04:43.981007 13266 raft_consensus.cc:2781] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:43.982251 13266 raft_consensus.cc:686] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [term 1 LEADER]: Becoming Leader. State: Replica: 1211bad1f809400e953f4d1642886b3d, State: Running, Role: LEADER
I20240502 14:04:43.982905 13266 consensus_queue.cc:237] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } }
I20240502 14:04:43.989703 12961 catalog_manager.cc:5484] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 1211bad1f809400e953f4d1642886b3d (127.11.9.66). New cstate: current_term: 1 leader_uuid: "1211bad1f809400e953f4d1642886b3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } health_report { overall_health: UNKNOWN } } }
I20240502 14:04:44.002720 13267 raft_consensus.cc:482] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:44.003089 13267 raft_consensus.cc:504] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } }
I20240502 14:04:44.004899 13267 leader_election.cc:288] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f41e9045e714e50af94553f17bb0ae3 (127.11.9.65:41711), 1211bad1f809400e953f4d1642886b3d (127.11.9.66:38837)
I20240502 14:04:44.014744 13065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ef8db613a46d98e647871c8180118" candidate_uuid: "c14a1e9fa43944fb952510b90e79bb30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f41e9045e714e50af94553f17bb0ae3" is_pre_election: true
I20240502 14:04:44.014897 13140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ef8db613a46d98e647871c8180118" candidate_uuid: "c14a1e9fa43944fb952510b90e79bb30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1211bad1f809400e953f4d1642886b3d" is_pre_election: true
I20240502 14:04:44.015265 13065 raft_consensus.cc:2454] T 192ef8db613a46d98e647871c8180118 P 8f41e9045e714e50af94553f17bb0ae3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c14a1e9fa43944fb952510b90e79bb30 in term 0.
I20240502 14:04:44.015432 13140 raft_consensus.cc:2454] T 192ef8db613a46d98e647871c8180118 P 1211bad1f809400e953f4d1642886b3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c14a1e9fa43944fb952510b90e79bb30 in term 0.
I20240502 14:04:44.016115 13179 leader_election.cc:302] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f41e9045e714e50af94553f17bb0ae3, c14a1e9fa43944fb952510b90e79bb30; no voters:
I20240502 14:04:44.016861 13267 raft_consensus.cc:2781] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:04:44.017171 13267 raft_consensus.cc:482] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:04:44.017377 13267 raft_consensus.cc:3037] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:44.021677 13267 raft_consensus.cc:504] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } }
I20240502 14:04:44.023048 13267 leader_election.cc:288] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30 [CANDIDATE]: Term 1 election: Requested vote from peers 8f41e9045e714e50af94553f17bb0ae3 (127.11.9.65:41711), 1211bad1f809400e953f4d1642886b3d (127.11.9.66:38837)
I20240502 14:04:44.023635 13065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ef8db613a46d98e647871c8180118" candidate_uuid: "c14a1e9fa43944fb952510b90e79bb30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f41e9045e714e50af94553f17bb0ae3"
I20240502 14:04:44.023974 13140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192ef8db613a46d98e647871c8180118" candidate_uuid: "c14a1e9fa43944fb952510b90e79bb30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1211bad1f809400e953f4d1642886b3d"
I20240502 14:04:44.024192 13065 raft_consensus.cc:3037] T 192ef8db613a46d98e647871c8180118 P 8f41e9045e714e50af94553f17bb0ae3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:44.024525 13140 raft_consensus.cc:3037] T 192ef8db613a46d98e647871c8180118 P 1211bad1f809400e953f4d1642886b3d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:04:44.030434 13065 raft_consensus.cc:2454] T 192ef8db613a46d98e647871c8180118 P 8f41e9045e714e50af94553f17bb0ae3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c14a1e9fa43944fb952510b90e79bb30 in term 1.
I20240502 14:04:44.030665 13140 raft_consensus.cc:2454] T 192ef8db613a46d98e647871c8180118 P 1211bad1f809400e953f4d1642886b3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c14a1e9fa43944fb952510b90e79bb30 in term 1.
I20240502 14:04:44.031407 13179 leader_election.cc:302] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f41e9045e714e50af94553f17bb0ae3, c14a1e9fa43944fb952510b90e79bb30; no voters:
I20240502 14:04:44.032068 13267 raft_consensus.cc:2781] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:04:44.033077 13267 raft_consensus.cc:686] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30 [term 1 LEADER]: Becoming Leader. State: Replica: c14a1e9fa43944fb952510b90e79bb30, State: Running, Role: LEADER
I20240502 14:04:44.033794 13267 consensus_queue.cc:237] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } }
I20240502 14:04:44.039615 12961 catalog_manager.cc:5484] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30 reported cstate change: term changed from 0 to 1, leader changed from <none> to c14a1e9fa43944fb952510b90e79bb30 (127.11.9.67). New cstate: current_term: 1 leader_uuid: "c14a1e9fa43944fb952510b90e79bb30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } health_report { overall_health: HEALTHY } } }
I20240502 14:04:44.290056 13016 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:04:44.291833 13016 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:04:44.292166 13016 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:04:44.500298 13279 consensus_queue.cc:1035] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:44.512601 13282 consensus_queue.cc:1035] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:44.615868 13267 consensus_queue.cc:1035] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:44.623840 13267 consensus_queue.cc:1035] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:04:46.293038 13016 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:04:46.295787 13016 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:04:46.296226 13016 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:04:48.297112 13016 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:04:48.299257 13016 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:04:48.299549 13016 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:04:50.300658 13016 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:04:50.302987 13016 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:04:50.303339 13016 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:04:52.304355 13016 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:04:52.306262 13016 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:04:52.306577 13016 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:04:54.307406 13016 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:04:54.309144 13016 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:04:54.309422 13016 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:04:55.120325 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:04:55.128193 13301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:55.128741 13302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:04:55.129659 13304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:04:55.130210 11301 server_base.cc:1034] running on GCE node
I20240502 14:04:55.132493 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:04:55.132686 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:04:55.132822 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658695132811 us; error 0 us; skew 500 ppm
I20240502 14:04:55.133272 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:04:55.140410 11301 webserver.cc:460] Webserver started at http://127.11.9.68:35667/ using document root <none> and password file <none>
I20240502 14:04:55.140872 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:04:55.141038 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:04:55.141307 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:04:55.142683 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658659912502-11301-0/minicluster-data/ts-3-root/instance:
uuid: "9d936791fa2547eb91c1ca5a27ed25fb"
format_stamp: "Formatted at 2024-05-02 14:04:55 on dist-test-slave-j72s"
I20240502 14:04:55.147492 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:04:55.151261 13309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:55.152104 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:04:55.152386 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658659912502-11301-0/minicluster-data/ts-3-root
uuid: "9d936791fa2547eb91c1ca5a27ed25fb"
format_stamp: "Formatted at 2024-05-02 14:04:55 on dist-test-slave-j72s"
I20240502 14:04:55.152639 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658659912502-11301-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658659912502-11301-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658659912502-11301-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:04:55.167100 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:04:55.168251 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:04:55.169932 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:04:55.172335 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:04:55.172506 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:55.172677 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:04:55.172830 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:04:55.213963 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.68:44549
I20240502 14:04:55.214061 13371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.68:44549 every 8 connection(s)
I20240502 14:04:55.228637 13372 heartbeater.cc:347] Connected to a master server at 127.11.9.126:40193
I20240502 14:04:55.228988 13372 heartbeater.cc:464] Registering TS with master...
I20240502 14:04:55.229679 13372 heartbeater.cc:511] Master 127.11.9.126:40193 requested a full tablet report, sending...
I20240502 14:04:55.231369 12962 ts_manager.cc:194] Registered new tserver with Master: 9d936791fa2547eb91c1ca5a27ed25fb (127.11.9.68:44549)
I20240502 14:04:55.232789 12962 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44358
I20240502 14:04:55.349056 13140 consensus_queue.cc:237] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "9d936791fa2547eb91c1ca5a27ed25fb" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 44549 } attrs { promote: true } }
I20240502 14:04:55.359416 13214 raft_consensus.cc:1261] T 4b48c7659b094e068c680f9b27fbcdee P c14a1e9fa43944fb952510b90e79bb30 [term 1 FOLLOWER]: Refusing update from remote peer 1211bad1f809400e953f4d1642886b3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:04:55.359606 13065 raft_consensus.cc:1261] T 4b48c7659b094e068c680f9b27fbcdee P 8f41e9045e714e50af94553f17bb0ae3 [term 1 FOLLOWER]: Refusing update from remote peer 1211bad1f809400e953f4d1642886b3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:04:55.361117 13300 consensus_queue.cc:1035] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 14:04:55.362232 13379 consensus_queue.cc:1035] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240502 14:04:55.370328 13105 consensus_peers.cc:485] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d -> Peer 9d936791fa2547eb91c1ca5a27ed25fb (127.11.9.68:44549): Couldn't send request to peer 9d936791fa2547eb91c1ca5a27ed25fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b48c7659b094e068c680f9b27fbcdee. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:04:55.371871 13379 raft_consensus.cc:2932] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9d936791fa2547eb91c1ca5a27ed25fb (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "9d936791fa2547eb91c1ca5a27ed25fb" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 44549 } attrs { promote: true } } }
I20240502 14:04:55.373401 13065 raft_consensus.cc:2932] T 4b48c7659b094e068c680f9b27fbcdee P 8f41e9045e714e50af94553f17bb0ae3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9d936791fa2547eb91c1ca5a27ed25fb (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "9d936791fa2547eb91c1ca5a27ed25fb" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 44549 } attrs { promote: true } } }
I20240502 14:04:55.377729 13214 raft_consensus.cc:2932] T 4b48c7659b094e068c680f9b27fbcdee P c14a1e9fa43944fb952510b90e79bb30 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9d936791fa2547eb91c1ca5a27ed25fb (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "9d936791fa2547eb91c1ca5a27ed25fb" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 44549 } attrs { promote: true } } }
I20240502 14:04:55.388206 12962 catalog_manager.cc:5484] T 4b48c7659b094e068c680f9b27fbcdee P 8f41e9045e714e50af94553f17bb0ae3 reported cstate change: config changed from index -1 to 2, NON_VOTER 9d936791fa2547eb91c1ca5a27ed25fb (127.11.9.68) added. New cstate: current_term: 1 leader_uuid: "1211bad1f809400e953f4d1642886b3d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "9d936791fa2547eb91c1ca5a27ed25fb" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 44549 } attrs { promote: true } } }
I20240502 14:04:55.775285 13390 ts_tablet_manager.cc:902] T 4b48c7659b094e068c680f9b27fbcdee P 9d936791fa2547eb91c1ca5a27ed25fb: Initiating tablet copy from peer 1211bad1f809400e953f4d1642886b3d (127.11.9.66:38837)
I20240502 14:04:55.779325 13390 tablet_copy_client.cc:308] T 4b48c7659b094e068c680f9b27fbcdee P 9d936791fa2547eb91c1ca5a27ed25fb: tablet copy: Beginning tablet copy session from remote peer at address 127.11.9.66:38837
I20240502 14:04:55.789578 13150 tablet_copy_service.cc:140] P 1211bad1f809400e953f4d1642886b3d: Received BeginTabletCopySession request for tablet 4b48c7659b094e068c680f9b27fbcdee from peer 9d936791fa2547eb91c1ca5a27ed25fb ({username='slave'} at 127.0.0.1:58686)
I20240502 14:04:55.790091 13150 tablet_copy_service.cc:161] P 1211bad1f809400e953f4d1642886b3d: Beginning new tablet copy session on tablet 4b48c7659b094e068c680f9b27fbcdee from peer 9d936791fa2547eb91c1ca5a27ed25fb at {username='slave'} at 127.0.0.1:58686: session id = 9d936791fa2547eb91c1ca5a27ed25fb-4b48c7659b094e068c680f9b27fbcdee
I20240502 14:04:55.797068 13150 tablet_copy_source_session.cc:205] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:04:55.799695 13390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b48c7659b094e068c680f9b27fbcdee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:04:55.811421 13390 tablet_copy_client.cc:783] T 4b48c7659b094e068c680f9b27fbcdee P 9d936791fa2547eb91c1ca5a27ed25fb: tablet copy: Starting download of 0 data blocks...
I20240502 14:04:55.811831 13390 tablet_copy_client.cc:647] T 4b48c7659b094e068c680f9b27fbcdee P 9d936791fa2547eb91c1ca5a27ed25fb: tablet copy: Starting download of 1 WAL segments...
I20240502 14:04:55.814822 13390 tablet_copy_client.cc:523] T 4b48c7659b094e068c680f9b27fbcdee P 9d936791fa2547eb91c1ca5a27ed25fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:04:55.821106 13390 tablet_bootstrap.cc:492] T 4b48c7659b094e068c680f9b27fbcdee P 9d936791fa2547eb91c1ca5a27ed25fb: Bootstrap starting.
I20240502 14:04:55.837292 13390 tablet_bootstrap.cc:492] T 4b48c7659b094e068c680f9b27fbcdee P 9d936791fa2547eb91c1ca5a27ed25fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:04:55.838021 13390 tablet_bootstrap.cc:492] T 4b48c7659b094e068c680f9b27fbcdee P 9d936791fa2547eb91c1ca5a27ed25fb: Bootstrap complete.
I20240502 14:04:55.838598 13390 ts_tablet_manager.cc:1367] T 4b48c7659b094e068c680f9b27fbcdee P 9d936791fa2547eb91c1ca5a27ed25fb: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.010s
I20240502 14:04:55.840938 13390 raft_consensus.cc:348] T 4b48c7659b094e068c680f9b27fbcdee P 9d936791fa2547eb91c1ca5a27ed25fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "9d936791fa2547eb91c1ca5a27ed25fb" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 44549 } attrs { promote: true } }
I20240502 14:04:55.841826 13390 raft_consensus.cc:729] T 4b48c7659b094e068c680f9b27fbcdee P 9d936791fa2547eb91c1ca5a27ed25fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9d936791fa2547eb91c1ca5a27ed25fb, State: Initialized, Role: LEARNER
I20240502 14:04:55.842335 13390 consensus_queue.cc:260] T 4b48c7659b094e068c680f9b27fbcdee P 9d936791fa2547eb91c1ca5a27ed25fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "9d936791fa2547eb91c1ca5a27ed25fb" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 44549 } attrs { promote: true } }
I20240502 14:04:55.844121 13372 heartbeater.cc:503] Master 127.11.9.126:40193 was elected leader, sending a full tablet report...
I20240502 14:04:55.844653 13390 ts_tablet_manager.cc:1398] T 4b48c7659b094e068c680f9b27fbcdee P 9d936791fa2547eb91c1ca5a27ed25fb: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240502 14:04:55.846176 13150 tablet_copy_service.cc:342] P 1211bad1f809400e953f4d1642886b3d: Request end of tablet copy session 9d936791fa2547eb91c1ca5a27ed25fb-4b48c7659b094e068c680f9b27fbcdee received from {username='slave'} at 127.0.0.1:58686
I20240502 14:04:55.846580 13150 tablet_copy_service.cc:433] P 1211bad1f809400e953f4d1642886b3d: ending tablet copy session 9d936791fa2547eb91c1ca5a27ed25fb-4b48c7659b094e068c680f9b27fbcdee on tablet 4b48c7659b094e068c680f9b27fbcdee with peer 9d936791fa2547eb91c1ca5a27ed25fb
I20240502 14:04:56.278771 13347 raft_consensus.cc:1203] T 4b48c7659b094e068c680f9b27fbcdee P 9d936791fa2547eb91c1ca5a27ed25fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:04:56.310240 13016 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:04:56.312500 13016 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:04:56.312923 13016 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:04:56.839737 13399 raft_consensus.cc:1050] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d: attempting to promote NON_VOTER 9d936791fa2547eb91c1ca5a27ed25fb to VOTER
I20240502 14:04:56.841671 13399 consensus_queue.cc:237] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "9d936791fa2547eb91c1ca5a27ed25fb" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44549 } attrs { promote: false } }
I20240502 14:04:56.847939 13347 raft_consensus.cc:1261] T 4b48c7659b094e068c680f9b27fbcdee P 9d936791fa2547eb91c1ca5a27ed25fb [term 1 LEARNER]: Refusing update from remote peer 1211bad1f809400e953f4d1642886b3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:04:56.849083 13214 raft_consensus.cc:1261] T 4b48c7659b094e068c680f9b27fbcdee P c14a1e9fa43944fb952510b90e79bb30 [term 1 FOLLOWER]: Refusing update from remote peer 1211bad1f809400e953f4d1642886b3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:04:56.849438 13065 raft_consensus.cc:1261] T 4b48c7659b094e068c680f9b27fbcdee P 8f41e9045e714e50af94553f17bb0ae3 [term 1 FOLLOWER]: Refusing update from remote peer 1211bad1f809400e953f4d1642886b3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:04:56.849614 13399 consensus_queue.cc:1035] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d936791fa2547eb91c1ca5a27ed25fb" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:04:56.850462 13400 consensus_queue.cc:1035] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:04:56.851454 13402 consensus_queue.cc:1035] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 14:04:56.862675 13400 raft_consensus.cc:2932] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9d936791fa2547eb91c1ca5a27ed25fb (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "9d936791fa2547eb91c1ca5a27ed25fb" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44549 } attrs { promote: false } } }
I20240502 14:04:56.864816 13347 raft_consensus.cc:2932] T 4b48c7659b094e068c680f9b27fbcdee P 9d936791fa2547eb91c1ca5a27ed25fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9d936791fa2547eb91c1ca5a27ed25fb (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "9d936791fa2547eb91c1ca5a27ed25fb" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44549 } attrs { promote: false } } }
I20240502 14:04:56.866497 13065 raft_consensus.cc:2932] T 4b48c7659b094e068c680f9b27fbcdee P 8f41e9045e714e50af94553f17bb0ae3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9d936791fa2547eb91c1ca5a27ed25fb (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "9d936791fa2547eb91c1ca5a27ed25fb" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44549 } attrs { promote: false } } }
I20240502 14:04:56.872577 13214 raft_consensus.cc:2932] T 4b48c7659b094e068c680f9b27fbcdee P c14a1e9fa43944fb952510b90e79bb30 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9d936791fa2547eb91c1ca5a27ed25fb (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "9d936791fa2547eb91c1ca5a27ed25fb" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44549 } attrs { promote: false } } }
I20240502 14:04:56.874928 12960 catalog_manager.cc:5484] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d reported cstate change: config changed from index 2 to 3, 9d936791fa2547eb91c1ca5a27ed25fb (127.11.9.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1211bad1f809400e953f4d1642886b3d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c14a1e9fa43944fb952510b90e79bb30" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34847 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d936791fa2547eb91c1ca5a27ed25fb" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44549 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:04:56.903765 13139 consensus_queue.cc:237] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "9d936791fa2547eb91c1ca5a27ed25fb" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44549 } attrs { promote: false } }
I20240502 14:04:56.908627 13347 raft_consensus.cc:1261] T 4b48c7659b094e068c680f9b27fbcdee P 9d936791fa2547eb91c1ca5a27ed25fb [term 1 FOLLOWER]: Refusing update from remote peer 1211bad1f809400e953f4d1642886b3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:04:56.908895 13065 raft_consensus.cc:1261] T 4b48c7659b094e068c680f9b27fbcdee P 8f41e9045e714e50af94553f17bb0ae3 [term 1 FOLLOWER]: Refusing update from remote peer 1211bad1f809400e953f4d1642886b3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:04:56.909885 13402 consensus_queue.cc:1035] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d936791fa2547eb91c1ca5a27ed25fb" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44549 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:04:56.910719 13399 consensus_queue.cc:1035] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:04:56.916500 13400 raft_consensus.cc:2932] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c14a1e9fa43944fb952510b90e79bb30 (127.11.9.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "9d936791fa2547eb91c1ca5a27ed25fb" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44549 } attrs { promote: false } } }
I20240502 14:04:56.918080 13347 raft_consensus.cc:2932] T 4b48c7659b094e068c680f9b27fbcdee P 9d936791fa2547eb91c1ca5a27ed25fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c14a1e9fa43944fb952510b90e79bb30 (127.11.9.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "9d936791fa2547eb91c1ca5a27ed25fb" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44549 } attrs { promote: false } } }
I20240502 14:04:56.918834 13065 raft_consensus.cc:2932] T 4b48c7659b094e068c680f9b27fbcdee P 8f41e9045e714e50af94553f17bb0ae3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c14a1e9fa43944fb952510b90e79bb30 (127.11.9.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "9d936791fa2547eb91c1ca5a27ed25fb" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44549 } attrs { promote: false } } }
I20240502 14:04:56.927175 12963 catalog_manager.cc:5484] T 4b48c7659b094e068c680f9b27fbcdee P 9d936791fa2547eb91c1ca5a27ed25fb reported cstate change: config changed from index 3 to 4, VOTER c14a1e9fa43944fb952510b90e79bb30 (127.11.9.67) evicted. New cstate: current_term: 1 leader_uuid: "1211bad1f809400e953f4d1642886b3d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1211bad1f809400e953f4d1642886b3d" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 38837 } } peers { permanent_uuid: "8f41e9045e714e50af94553f17bb0ae3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 41711 } } peers { permanent_uuid: "9d936791fa2547eb91c1ca5a27ed25fb" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44549 } attrs { promote: false } } }
I20240502 14:04:56.928889 12948 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4b48c7659b094e068c680f9b27fbcdee with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:04:56.937485 13203 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b48c7659b094e068c680f9b27fbcdee with delete_type TABLET_DATA_TOMBSTONED (TS c14a1e9fa43944fb952510b90e79bb30 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51962
I20240502 14:04:56.939754 13412 tablet_replica.cc:330] T 4b48c7659b094e068c680f9b27fbcdee P c14a1e9fa43944fb952510b90e79bb30: stopping tablet replica
I20240502 14:04:56.940423 13412 raft_consensus.cc:2229] T 4b48c7659b094e068c680f9b27fbcdee P c14a1e9fa43944fb952510b90e79bb30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:04:56.941059 13412 raft_consensus.cc:2258] T 4b48c7659b094e068c680f9b27fbcdee P c14a1e9fa43944fb952510b90e79bb30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:04:56.944248 13412 ts_tablet_manager.cc:1875] T 4b48c7659b094e068c680f9b27fbcdee P c14a1e9fa43944fb952510b90e79bb30: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:04:56.953624 13412 ts_tablet_manager.cc:1888] T 4b48c7659b094e068c680f9b27fbcdee P c14a1e9fa43944fb952510b90e79bb30: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:04:56.953943 13412 log.cc:1196] T 4b48c7659b094e068c680f9b27fbcdee P c14a1e9fa43944fb952510b90e79bb30: Deleting WAL directory at /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714658659912502-11301-0/minicluster-data/ts-2-root/wals/4b48c7659b094e068c680f9b27fbcdee
I20240502 14:04:56.955292 12949 catalog_manager.cc:4830] TS c14a1e9fa43944fb952510b90e79bb30 (127.11.9.67:34847): tablet 4b48c7659b094e068c680f9b27fbcdee (table test-workload [id=fd4179708e91461e8b7e0832a19c11ac]) successfully deleted
I20240502 14:04:58.313971 13016 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:04:58.315672 13016 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:04:58.315917 13016 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:00.317313 13016 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:00.319801 13016 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:00.320080 13016 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:02.321084 13016 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:02.323047 13016 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:02.323356 13016 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:04.324163 13016 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:04.325920 13016 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:04.326206 13016 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:06.327072 13016 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:06.328960 13016 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:06.329272 13016 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:08.330188 13016 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:08.332029 13016 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:08.332340 13016 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:09.058400 11301 tablet_server.cc:178] TabletServer@127.11.9.65:0 shutting down...
I20240502 14:05:09.083516 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:05:09.084326 11301 tablet_replica.cc:330] T 4b48c7659b094e068c680f9b27fbcdee P 8f41e9045e714e50af94553f17bb0ae3: stopping tablet replica
I20240502 14:05:09.085158 11301 raft_consensus.cc:2229] T 4b48c7659b094e068c680f9b27fbcdee P 8f41e9045e714e50af94553f17bb0ae3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:09.086217 11301 raft_consensus.cc:2258] T 4b48c7659b094e068c680f9b27fbcdee P 8f41e9045e714e50af94553f17bb0ae3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:09.088347 11301 tablet_replica.cc:330] T 192ef8db613a46d98e647871c8180118 P 8f41e9045e714e50af94553f17bb0ae3: stopping tablet replica
I20240502 14:05:09.088781 11301 raft_consensus.cc:2229] T 192ef8db613a46d98e647871c8180118 P 8f41e9045e714e50af94553f17bb0ae3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:09.089175 11301 raft_consensus.cc:2258] T 192ef8db613a46d98e647871c8180118 P 8f41e9045e714e50af94553f17bb0ae3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:09.110360 11301 tablet_server.cc:194] TabletServer@127.11.9.65:0 shutdown complete.
W20240502 14:05:09.118480 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.9.65:41711: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240502 14:05:09.122272 13105 consensus_peers.cc:485] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d -> Peer 8f41e9045e714e50af94553f17bb0ae3 (127.11.9.65:41711): Couldn't send request to peer 8f41e9045e714e50af94553f17bb0ae3. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.65:41711: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:09.124449 11301 tablet_server.cc:178] TabletServer@127.11.9.66:0 shutting down...
I20240502 14:05:09.146234 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:05:09.146802 11301 tablet_replica.cc:330] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d: stopping tablet replica
I20240502 14:05:09.147352 11301 raft_consensus.cc:2229] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:05:09.148128 11301 raft_consensus.cc:2258] T 4b48c7659b094e068c680f9b27fbcdee P 1211bad1f809400e953f4d1642886b3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:09.149986 11301 tablet_replica.cc:330] T 192ef8db613a46d98e647871c8180118 P 1211bad1f809400e953f4d1642886b3d: stopping tablet replica
I20240502 14:05:09.150370 11301 raft_consensus.cc:2229] T 192ef8db613a46d98e647871c8180118 P 1211bad1f809400e953f4d1642886b3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:09.150705 11301 raft_consensus.cc:2258] T 192ef8db613a46d98e647871c8180118 P 1211bad1f809400e953f4d1642886b3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:09.172158 11301 tablet_server.cc:194] TabletServer@127.11.9.66:0 shutdown complete.
I20240502 14:05:09.184094 11301 tablet_server.cc:178] TabletServer@127.11.9.67:0 shutting down...
I20240502 14:05:09.208432 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:05:09.209107 11301 tablet_replica.cc:330] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30: stopping tablet replica
I20240502 14:05:09.209686 11301 raft_consensus.cc:2229] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:05:09.210489 11301 raft_consensus.cc:2258] T 192ef8db613a46d98e647871c8180118 P c14a1e9fa43944fb952510b90e79bb30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:09.230541 11301 tablet_server.cc:194] TabletServer@127.11.9.67:0 shutdown complete.
I20240502 14:05:09.240809 11301 tablet_server.cc:178] TabletServer@127.11.9.68:0 shutting down...
I20240502 14:05:09.260610 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:05:09.261265 11301 tablet_replica.cc:330] T 4b48c7659b094e068c680f9b27fbcdee P 9d936791fa2547eb91c1ca5a27ed25fb: stopping tablet replica
I20240502 14:05:09.261798 11301 raft_consensus.cc:2229] T 4b48c7659b094e068c680f9b27fbcdee P 9d936791fa2547eb91c1ca5a27ed25fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:09.262382 11301 raft_consensus.cc:2258] T 4b48c7659b094e068c680f9b27fbcdee P 9d936791fa2547eb91c1ca5a27ed25fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:09.280366 11301 tablet_server.cc:194] TabletServer@127.11.9.68:0 shutdown complete.
I20240502 14:05:09.289786 11301 master.cc:537] Master@127.11.9.126:40193 shutting down...
I20240502 14:05:09.306089 11301 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5c48abe57dc8447d8b6d4bed9d17be45 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:05:09.306596 11301 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5c48abe57dc8447d8b6d4bed9d17be45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:09.306862 11301 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5c48abe57dc8447d8b6d4bed9d17be45: stopping tablet replica
I20240502 14:05:09.324184 11301 master.cc:552] Master@127.11.9.126:40193 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (27217 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240502 14:05:09.358191 11301 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.9.126:44113
I20240502 14:05:09.359252 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:09.364825 13421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:09.365638 13422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:09.366243 13424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:09.366938 11301 server_base.cc:1034] running on GCE node
I20240502 14:05:09.367782 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:05:09.367972 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:05:09.368129 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658709368112 us; error 0 us; skew 500 ppm
I20240502 14:05:09.368593 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:09.370795 11301 webserver.cc:460] Webserver started at http://127.11.9.126:44739/ using document root <none> and password file <none>
I20240502 14:05:09.371194 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:09.371351 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:09.371584 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:09.372735 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714658659912502-11301-0/minicluster-data/master-0-root/instance:
uuid: "323dd00ba494455a96db47ab2243b236"
format_stamp: "Formatted at 2024-05-02 14:05:09 on dist-test-slave-j72s"
I20240502 14:05:09.376637 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240502 14:05:09.379565 13429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:09.380182 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 14:05:09.380429 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714658659912502-11301-0/minicluster-data/master-0-root
uuid: "323dd00ba494455a96db47ab2243b236"
format_stamp: "Formatted at 2024-05-02 14:05:09 on dist-test-slave-j72s"
I20240502 14:05:09.380672 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714658659912502-11301-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714658659912502-11301-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714658659912502-11301-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:09.394788 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:09.395828 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:09.429627 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.126:44113
I20240502 14:05:09.429714 13480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.126:44113 every 8 connection(s)
I20240502 14:05:09.433332 13481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:09.443365 13481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 323dd00ba494455a96db47ab2243b236: Bootstrap starting.
I20240502 14:05:09.447230 13481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 323dd00ba494455a96db47ab2243b236: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:09.451021 13481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 323dd00ba494455a96db47ab2243b236: No bootstrap required, opened a new log
I20240502 14:05:09.452961 13481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 323dd00ba494455a96db47ab2243b236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323dd00ba494455a96db47ab2243b236" member_type: VOTER }
I20240502 14:05:09.453425 13481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 323dd00ba494455a96db47ab2243b236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:09.453656 13481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 323dd00ba494455a96db47ab2243b236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 323dd00ba494455a96db47ab2243b236, State: Initialized, Role: FOLLOWER
I20240502 14:05:09.454149 13481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 323dd00ba494455a96db47ab2243b236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323dd00ba494455a96db47ab2243b236" member_type: VOTER }
I20240502 14:05:09.454555 13481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 323dd00ba494455a96db47ab2243b236 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:05:09.454731 13481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 323dd00ba494455a96db47ab2243b236 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:05:09.454923 13481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 323dd00ba494455a96db47ab2243b236 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:09.458806 13481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 323dd00ba494455a96db47ab2243b236 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323dd00ba494455a96db47ab2243b236" member_type: VOTER }
I20240502 14:05:09.459235 13481 leader_election.cc:302] T 00000000000000000000000000000000 P 323dd00ba494455a96db47ab2243b236 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 323dd00ba494455a96db47ab2243b236; no voters:
I20240502 14:05:09.460228 13481 leader_election.cc:288] T 00000000000000000000000000000000 P 323dd00ba494455a96db47ab2243b236 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:05:09.460613 13484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 323dd00ba494455a96db47ab2243b236 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:09.462185 13484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 323dd00ba494455a96db47ab2243b236 [term 1 LEADER]: Becoming Leader. State: Replica: 323dd00ba494455a96db47ab2243b236, State: Running, Role: LEADER
I20240502 14:05:09.463064 13481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 323dd00ba494455a96db47ab2243b236 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:05:09.462926 13484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 323dd00ba494455a96db47ab2243b236 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323dd00ba494455a96db47ab2243b236" member_type: VOTER }
I20240502 14:05:09.468315 13486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323dd00ba494455a96db47ab2243b236 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 323dd00ba494455a96db47ab2243b236. Latest consensus state: current_term: 1 leader_uuid: "323dd00ba494455a96db47ab2243b236" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323dd00ba494455a96db47ab2243b236" member_type: VOTER } }
I20240502 14:05:09.469681 13486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323dd00ba494455a96db47ab2243b236 [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:09.469491 13485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323dd00ba494455a96db47ab2243b236 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "323dd00ba494455a96db47ab2243b236" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323dd00ba494455a96db47ab2243b236" member_type: VOTER } }
I20240502 14:05:09.470276 13485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323dd00ba494455a96db47ab2243b236 [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:09.472664 13494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:05:09.477421 13494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:05:09.480531 11301 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:05:09.485574 13494 catalog_manager.cc:1347] Generated new cluster ID: b54c4d04f852473cbfe24e582d44d534
I20240502 14:05:09.485888 13494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:05:09.499754 13494 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:05:09.501013 13494 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:05:09.523244 13494 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 323dd00ba494455a96db47ab2243b236: Generated new TSK 0
I20240502 14:05:09.523788 13494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:05:09.546180 11301 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 0.000190529s
I20240502 14:05:21.639616 11301 master.cc:537] Master@127.11.9.126:44113 shutting down...
I20240502 14:05:21.660748 11301 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 323dd00ba494455a96db47ab2243b236 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:05:21.661562 11301 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 323dd00ba494455a96db47ab2243b236 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:21.661911 11301 tablet_replica.cc:330] T 00000000000000000000000000000000 P 323dd00ba494455a96db47ab2243b236: stopping tablet replica
I20240502 14:05:21.680050 11301 master.cc:552] Master@127.11.9.126:44113 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12350 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240502 14:05:21.708899 11301 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.9.126:34501
I20240502 14:05:21.709964 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:21.715803 13505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:21.716425 13506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:21.717200 13508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:21.717859 11301 server_base.cc:1034] running on GCE node
I20240502 14:05:21.718480 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:05:21.718652 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:05:21.718787 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658721718771 us; error 0 us; skew 500 ppm
I20240502 14:05:21.719264 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:21.729935 11301 webserver.cc:460] Webserver started at http://127.11.9.126:43679/ using document root <none> and password file <none>
I20240502 14:05:21.730381 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:21.730540 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:21.730788 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:21.731926 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658659912502-11301-0/minicluster-data/master-0-root/instance:
uuid: "a3b7b6ca7f804a08a7ff0865963b9764"
format_stamp: "Formatted at 2024-05-02 14:05:21 on dist-test-slave-j72s"
I20240502 14:05:21.736328 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 14:05:21.739410 13513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:21.740046 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:05:21.740314 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658659912502-11301-0/minicluster-data/master-0-root
uuid: "a3b7b6ca7f804a08a7ff0865963b9764"
format_stamp: "Formatted at 2024-05-02 14:05:21 on dist-test-slave-j72s"
I20240502 14:05:21.740562 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658659912502-11301-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658659912502-11301-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658659912502-11301-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:21.754829 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:21.755957 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:21.791752 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.126:34501
I20240502 14:05:21.791857 13564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.126:34501 every 8 connection(s)
I20240502 14:05:21.795599 13565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:21.805593 13565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3b7b6ca7f804a08a7ff0865963b9764: Bootstrap starting.
I20240502 14:05:21.809336 13565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3b7b6ca7f804a08a7ff0865963b9764: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:21.813190 13565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3b7b6ca7f804a08a7ff0865963b9764: No bootstrap required, opened a new log
I20240502 14:05:21.815069 13565 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3b7b6ca7f804a08a7ff0865963b9764 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b7b6ca7f804a08a7ff0865963b9764" member_type: VOTER }
I20240502 14:05:21.815515 13565 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3b7b6ca7f804a08a7ff0865963b9764 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:21.815696 13565 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3b7b6ca7f804a08a7ff0865963b9764 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3b7b6ca7f804a08a7ff0865963b9764, State: Initialized, Role: FOLLOWER
I20240502 14:05:21.816161 13565 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3b7b6ca7f804a08a7ff0865963b9764 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b7b6ca7f804a08a7ff0865963b9764" member_type: VOTER }
I20240502 14:05:21.816591 13565 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3b7b6ca7f804a08a7ff0865963b9764 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:05:21.816766 13565 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3b7b6ca7f804a08a7ff0865963b9764 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:05:21.816972 13565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3b7b6ca7f804a08a7ff0865963b9764 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:21.820883 13565 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3b7b6ca7f804a08a7ff0865963b9764 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b7b6ca7f804a08a7ff0865963b9764" member_type: VOTER }
I20240502 14:05:21.821298 13565 leader_election.cc:302] T 00000000000000000000000000000000 P a3b7b6ca7f804a08a7ff0865963b9764 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3b7b6ca7f804a08a7ff0865963b9764; no voters:
I20240502 14:05:21.822280 13565 leader_election.cc:288] T 00000000000000000000000000000000 P a3b7b6ca7f804a08a7ff0865963b9764 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:05:21.822566 13568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3b7b6ca7f804a08a7ff0865963b9764 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:21.823799 13568 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3b7b6ca7f804a08a7ff0865963b9764 [term 1 LEADER]: Becoming Leader. State: Replica: a3b7b6ca7f804a08a7ff0865963b9764, State: Running, Role: LEADER
I20240502 14:05:21.824437 13568 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3b7b6ca7f804a08a7ff0865963b9764 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b7b6ca7f804a08a7ff0865963b9764" member_type: VOTER }
I20240502 14:05:21.825003 13565 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3b7b6ca7f804a08a7ff0865963b9764 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:05:21.827324 13569 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3b7b6ca7f804a08a7ff0865963b9764 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3b7b6ca7f804a08a7ff0865963b9764" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b7b6ca7f804a08a7ff0865963b9764" member_type: VOTER } }
I20240502 14:05:21.827415 13570 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3b7b6ca7f804a08a7ff0865963b9764 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3b7b6ca7f804a08a7ff0865963b9764. Latest consensus state: current_term: 1 leader_uuid: "a3b7b6ca7f804a08a7ff0865963b9764" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b7b6ca7f804a08a7ff0865963b9764" member_type: VOTER } }
I20240502 14:05:21.828133 13569 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3b7b6ca7f804a08a7ff0865963b9764 [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:21.828922 13570 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3b7b6ca7f804a08a7ff0865963b9764 [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:21.831456 13575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:05:21.835870 13575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:05:21.839752 11301 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:05:21.844098 13575 catalog_manager.cc:1347] Generated new cluster ID: c4221b040f524080b0d905d00c97e01e
I20240502 14:05:21.844300 13575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:05:21.867419 13575 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:05:21.868557 13575 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:05:21.886643 13575 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a3b7b6ca7f804a08a7ff0865963b9764: Generated new TSK 0
I20240502 14:05:21.887135 13575 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:05:21.905725 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:21.912387 13586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:21.913540 13587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:21.914448 13589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:21.915275 11301 server_base.cc:1034] running on GCE node
I20240502 14:05:21.916158 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:05:21.916363 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:05:21.916532 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658721916516 us; error 0 us; skew 500 ppm
I20240502 14:05:21.917066 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:21.919708 11301 webserver.cc:460] Webserver started at http://127.11.9.65:39213/ using document root <none> and password file <none>
I20240502 14:05:21.920153 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:21.920359 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:21.920600 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:21.921841 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658659912502-11301-0/minicluster-data/ts-0-root/instance:
uuid: "fa2c249676f341df8063de316f479039"
format_stamp: "Formatted at 2024-05-02 14:05:21 on dist-test-slave-j72s"
I20240502 14:05:21.926775 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240502 14:05:21.929858 13594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:21.930575 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240502 14:05:21.930830 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658659912502-11301-0/minicluster-data/ts-0-root
uuid: "fa2c249676f341df8063de316f479039"
format_stamp: "Formatted at 2024-05-02 14:05:21 on dist-test-slave-j72s"
I20240502 14:05:21.931092 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658659912502-11301-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658659912502-11301-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658659912502-11301-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:21.943962 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:21.944976 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:21.946355 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:21.948474 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:21.948658 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:21.948868 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:21.949033 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:21.997479 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.65:38379
I20240502 14:05:21.997603 13656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.65:38379 every 8 connection(s)
I20240502 14:05:22.002079 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:22.010435 13661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:22.011883 13662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:22.014410 11301 server_base.cc:1034] running on GCE node
W20240502 14:05:22.014956 13664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:22.015197 13657 heartbeater.cc:347] Connected to a master server at 127.11.9.126:34501
I20240502 14:05:22.015594 13657 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:22.015893 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:05:22.016124 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:05:22.016319 13657 heartbeater.cc:511] Master 127.11.9.126:34501 requested a full tablet report, sending...
I20240502 14:05:22.016279 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658722016259 us; error 0 us; skew 500 ppm
I20240502 14:05:22.017016 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:22.018513 13530 ts_manager.cc:194] Registered new tserver with Master: fa2c249676f341df8063de316f479039 (127.11.9.65:38379)
I20240502 14:05:22.019834 11301 webserver.cc:460] Webserver started at http://127.11.9.66:37059/ using document root <none> and password file <none>
I20240502 14:05:22.020464 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:22.020702 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:22.020999 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:22.021204 13530 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53304
I20240502 14:05:22.022621 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658659912502-11301-0/minicluster-data/ts-1-root/instance:
uuid: "c5d85a39c6b2487a9146b8465a1b02e7"
format_stamp: "Formatted at 2024-05-02 14:05:22 on dist-test-slave-j72s"
I20240502 14:05:22.026939 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20240502 14:05:22.030019 13669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:22.030740 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240502 14:05:22.031015 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658659912502-11301-0/minicluster-data/ts-1-root
uuid: "c5d85a39c6b2487a9146b8465a1b02e7"
format_stamp: "Formatted at 2024-05-02 14:05:22 on dist-test-slave-j72s"
I20240502 14:05:22.031263 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658659912502-11301-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658659912502-11301-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658659912502-11301-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:22.043916 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:22.044992 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:22.046303 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:22.048385 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:22.048573 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:22.048785 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:22.048935 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:22.086905 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.66:44575
I20240502 14:05:22.087011 13731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.66:44575 every 8 connection(s)
I20240502 14:05:22.091197 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:22.098722 13735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:22.099452 13736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:22.101894 11301 server_base.cc:1034] running on GCE node
W20240502 14:05:22.102638 13738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:22.103017 13732 heartbeater.cc:347] Connected to a master server at 127.11.9.126:34501
I20240502 14:05:22.103363 13732 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:22.103525 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:05:22.103749 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:05:22.103906 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658722103889 us; error 0 us; skew 500 ppm
I20240502 14:05:22.104014 13732 heartbeater.cc:511] Master 127.11.9.126:34501 requested a full tablet report, sending...
I20240502 14:05:22.104436 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:22.105953 13530 ts_manager.cc:194] Registered new tserver with Master: c5d85a39c6b2487a9146b8465a1b02e7 (127.11.9.66:44575)
I20240502 14:05:22.107120 11301 webserver.cc:460] Webserver started at http://127.11.9.67:33857/ using document root <none> and password file <none>
I20240502 14:05:22.107403 13530 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53318
I20240502 14:05:22.107555 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:22.107820 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:22.108074 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:22.109320 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658659912502-11301-0/minicluster-data/ts-2-root/instance:
uuid: "15828f9490af46e3ac2ecee55f5162b9"
format_stamp: "Formatted at 2024-05-02 14:05:22 on dist-test-slave-j72s"
I20240502 14:05:22.113330 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240502 14:05:22.116258 13743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:22.116917 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:05:22.117139 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658659912502-11301-0/minicluster-data/ts-2-root
uuid: "15828f9490af46e3ac2ecee55f5162b9"
format_stamp: "Formatted at 2024-05-02 14:05:22 on dist-test-slave-j72s"
I20240502 14:05:22.117363 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658659912502-11301-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658659912502-11301-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714658659912502-11301-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:22.141595 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:22.142699 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:22.144009 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:22.146034 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:22.146242 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:22.146445 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:22.146590 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:22.184002 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.67:39667
I20240502 14:05:22.184096 13805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.67:39667 every 8 connection(s)
I20240502 14:05:22.196671 13806 heartbeater.cc:347] Connected to a master server at 127.11.9.126:34501
I20240502 14:05:22.197059 13806 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:22.197778 13806 heartbeater.cc:511] Master 127.11.9.126:34501 requested a full tablet report, sending...
I20240502 14:05:22.199482 13530 ts_manager.cc:194] Registered new tserver with Master: 15828f9490af46e3ac2ecee55f5162b9 (127.11.9.67:39667)
I20240502 14:05:22.200524 11301 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013511631s
I20240502 14:05:22.200898 13530 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53324
I20240502 14:05:23.023836 13657 heartbeater.cc:503] Master 127.11.9.126:34501 was elected leader, sending a full tablet report...
I20240502 14:05:23.109627 13732 heartbeater.cc:503] Master 127.11.9.126:34501 was elected leader, sending a full tablet report...
I20240502 14:05:23.203286 13806 heartbeater.cc:503] Master 127.11.9.126:34501 was elected leader, sending a full tablet report...
I20240502 14:05:23.839298 13583 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:25.840202 13583 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:27.841157 13583 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:29.842115 13583 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:31.843165 13583 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:33.844488 13583 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:34.294427 11301 tablet_server.cc:178] TabletServer@127.11.9.65:0 shutting down...
I20240502 14:05:34.316288 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:05:34.332240 11301 tablet_server.cc:194] TabletServer@127.11.9.65:0 shutdown complete.
I20240502 14:05:34.341636 11301 tablet_server.cc:178] TabletServer@127.11.9.66:0 shutting down...
I20240502 14:05:34.363163 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:05:34.378221 11301 tablet_server.cc:194] TabletServer@127.11.9.66:0 shutdown complete.
I20240502 14:05:34.386031 11301 tablet_server.cc:178] TabletServer@127.11.9.67:0 shutting down...
I20240502 14:05:34.404202 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:05:34.419346 11301 tablet_server.cc:194] TabletServer@127.11.9.67:0 shutdown complete.
I20240502 14:05:34.427320 11301 master.cc:537] Master@127.11.9.126:34501 shutting down...
I20240502 14:05:34.443768 11301 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a3b7b6ca7f804a08a7ff0865963b9764 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:05:34.444504 11301 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a3b7b6ca7f804a08a7ff0865963b9764 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:34.444859 11301 tablet_replica.cc:330] T 00000000000000000000000000000000 P a3b7b6ca7f804a08a7ff0865963b9764: stopping tablet replica
I20240502 14:05:34.462241 11301 master.cc:552] Master@127.11.9.126:34501 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12780 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240502 14:05:34.489693 11301 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.9.126:38683
I20240502 14:05:34.490674 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:34.496294 13811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:34.496899 13812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:34.497320 13814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:34.498057 11301 server_base.cc:1034] running on GCE node
I20240502 14:05:34.498715 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:05:34.498888 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:05:34.499029 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658734499011 us; error 0 us; skew 500 ppm
I20240502 14:05:34.499442 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:34.507328 11301 webserver.cc:460] Webserver started at http://127.11.9.126:44493/ using document root <none> and password file <none>
I20240502 14:05:34.507750 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:34.507908 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:34.508150 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:34.509219 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658659912502-11301-0/minicluster-data/master-0-root/instance:
uuid: "8258f89f0a6b400a82af70c742262a67"
format_stamp: "Formatted at 2024-05-02 14:05:34 on dist-test-slave-j72s"
I20240502 14:05:34.513346 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 14:05:34.516225 13819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:34.516860 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:05:34.517076 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658659912502-11301-0/minicluster-data/master-0-root
uuid: "8258f89f0a6b400a82af70c742262a67"
format_stamp: "Formatted at 2024-05-02 14:05:34 on dist-test-slave-j72s"
I20240502 14:05:34.517313 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658659912502-11301-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658659912502-11301-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658659912502-11301-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:34.532320 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:34.533336 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:34.567092 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.126:38683
I20240502 14:05:34.567184 13870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.126:38683 every 8 connection(s)
I20240502 14:05:34.570617 13871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:34.580935 13871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8258f89f0a6b400a82af70c742262a67: Bootstrap starting.
I20240502 14:05:34.584880 13871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8258f89f0a6b400a82af70c742262a67: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:34.588546 13871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8258f89f0a6b400a82af70c742262a67: No bootstrap required, opened a new log
I20240502 14:05:34.590384 13871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8258f89f0a6b400a82af70c742262a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8258f89f0a6b400a82af70c742262a67" member_type: VOTER }
I20240502 14:05:34.590845 13871 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8258f89f0a6b400a82af70c742262a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:34.591024 13871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8258f89f0a6b400a82af70c742262a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8258f89f0a6b400a82af70c742262a67, State: Initialized, Role: FOLLOWER
I20240502 14:05:34.591532 13871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8258f89f0a6b400a82af70c742262a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8258f89f0a6b400a82af70c742262a67" member_type: VOTER }
I20240502 14:05:34.591969 13871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8258f89f0a6b400a82af70c742262a67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:05:34.592221 13871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8258f89f0a6b400a82af70c742262a67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:05:34.592444 13871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8258f89f0a6b400a82af70c742262a67 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:34.596143 13871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8258f89f0a6b400a82af70c742262a67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8258f89f0a6b400a82af70c742262a67" member_type: VOTER }
I20240502 14:05:34.596571 13871 leader_election.cc:302] T 00000000000000000000000000000000 P 8258f89f0a6b400a82af70c742262a67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8258f89f0a6b400a82af70c742262a67; no voters:
I20240502 14:05:34.597602 13871 leader_election.cc:288] T 00000000000000000000000000000000 P 8258f89f0a6b400a82af70c742262a67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:05:34.597932 13874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8258f89f0a6b400a82af70c742262a67 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:34.599169 13874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8258f89f0a6b400a82af70c742262a67 [term 1 LEADER]: Becoming Leader. State: Replica: 8258f89f0a6b400a82af70c742262a67, State: Running, Role: LEADER
I20240502 14:05:34.599851 13874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8258f89f0a6b400a82af70c742262a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8258f89f0a6b400a82af70c742262a67" member_type: VOTER }
I20240502 14:05:34.600392 13871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8258f89f0a6b400a82af70c742262a67 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:05:34.602703 13876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8258f89f0a6b400a82af70c742262a67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8258f89f0a6b400a82af70c742262a67. Latest consensus state: current_term: 1 leader_uuid: "8258f89f0a6b400a82af70c742262a67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8258f89f0a6b400a82af70c742262a67" member_type: VOTER } }
I20240502 14:05:34.602666 13875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8258f89f0a6b400a82af70c742262a67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8258f89f0a6b400a82af70c742262a67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8258f89f0a6b400a82af70c742262a67" member_type: VOTER } }
I20240502 14:05:34.603283 13876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8258f89f0a6b400a82af70c742262a67 [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:34.603370 13875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8258f89f0a6b400a82af70c742262a67 [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:34.606935 13879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:05:34.611120 13879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:05:34.615293 11301 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:05:34.619626 13879 catalog_manager.cc:1347] Generated new cluster ID: 459fffffed0e49a195be8cc89b38077b
I20240502 14:05:34.619824 13879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:05:34.629493 13879 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:05:34.630525 13879 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:05:34.641961 13879 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8258f89f0a6b400a82af70c742262a67: Generated new TSK 0
I20240502 14:05:34.642400 13879 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:05:34.648950 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:34.654772 13892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:34.655704 13893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:34.657239 13895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:34.658072 11301 server_base.cc:1034] running on GCE node
I20240502 14:05:34.658785 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:05:34.658957 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:05:34.659067 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658734659056 us; error 0 us; skew 500 ppm
I20240502 14:05:34.659454 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:34.661588 11301 webserver.cc:460] Webserver started at http://127.11.9.65:37781/ using document root <none> and password file <none>
I20240502 14:05:34.661959 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:34.662104 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:34.662287 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:34.663234 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658659912502-11301-0/minicluster-data/ts-0-root/instance:
uuid: "4b7709fd386845c5b1b11e0f80b40ee6"
format_stamp: "Formatted at 2024-05-02 14:05:34 on dist-test-slave-j72s"
I20240502 14:05:34.667423 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:05:34.670207 13900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:34.670897 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:05:34.671121 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658659912502-11301-0/minicluster-data/ts-0-root
uuid: "4b7709fd386845c5b1b11e0f80b40ee6"
format_stamp: "Formatted at 2024-05-02 14:05:34 on dist-test-slave-j72s"
I20240502 14:05:34.671348 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658659912502-11301-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658659912502-11301-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658659912502-11301-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:34.685379 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:34.686398 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:34.687707 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:34.689849 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:34.690022 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:34.690264 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:34.690415 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:34.728607 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.65:42985
I20240502 14:05:34.728734 13962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.65:42985 every 8 connection(s)
I20240502 14:05:34.733009 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:34.741252 13967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:34.742290 13968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:34.745301 13970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:34.745826 13963 heartbeater.cc:347] Connected to a master server at 127.11.9.126:38683
I20240502 14:05:34.746098 13963 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:34.746594 11301 server_base.cc:1034] running on GCE node
I20240502 14:05:34.746749 13963 heartbeater.cc:511] Master 127.11.9.126:38683 requested a full tablet report, sending...
I20240502 14:05:34.747387 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:05:34.747594 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:05:34.747747 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658734747731 us; error 0 us; skew 500 ppm
I20240502 14:05:34.748298 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:34.748622 13836 ts_manager.cc:194] Registered new tserver with Master: 4b7709fd386845c5b1b11e0f80b40ee6 (127.11.9.65:42985)
I20240502 14:05:34.750613 13836 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33042
I20240502 14:05:34.750777 11301 webserver.cc:460] Webserver started at http://127.11.9.66:46487/ using document root <none> and password file <none>
I20240502 14:05:34.751298 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:34.751482 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:34.751727 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:34.752872 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658659912502-11301-0/minicluster-data/ts-1-root/instance:
uuid: "74ceb9c7567f4dd58acfc46ba1b45000"
format_stamp: "Formatted at 2024-05-02 14:05:34 on dist-test-slave-j72s"
I20240502 14:05:34.756915 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 14:05:34.759936 13975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:34.760584 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:05:34.760840 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658659912502-11301-0/minicluster-data/ts-1-root
uuid: "74ceb9c7567f4dd58acfc46ba1b45000"
format_stamp: "Formatted at 2024-05-02 14:05:34 on dist-test-slave-j72s"
I20240502 14:05:34.761103 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658659912502-11301-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658659912502-11301-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658659912502-11301-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:34.779814 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:34.780884 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:34.782208 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:34.784271 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:34.784456 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:34.784663 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:34.784812 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:34.822724 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.66:33011
I20240502 14:05:34.822810 14037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.66:33011 every 8 connection(s)
I20240502 14:05:34.827085 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:34.833973 14041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:34.834909 14042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:34.836652 14044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:34.837277 11301 server_base.cc:1034] running on GCE node
I20240502 14:05:34.838155 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:05:34.838402 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:05:34.838600 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658734838582 us; error 0 us; skew 500 ppm
I20240502 14:05:34.839265 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:34.839522 14038 heartbeater.cc:347] Connected to a master server at 127.11.9.126:38683
I20240502 14:05:34.839897 14038 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:34.840754 14038 heartbeater.cc:511] Master 127.11.9.126:38683 requested a full tablet report, sending...
I20240502 14:05:34.841800 11301 webserver.cc:460] Webserver started at http://127.11.9.67:44733/ using document root <none> and password file <none>
I20240502 14:05:34.842392 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:34.842624 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:34.842728 13836 ts_manager.cc:194] Registered new tserver with Master: 74ceb9c7567f4dd58acfc46ba1b45000 (127.11.9.66:33011)
I20240502 14:05:34.842983 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:34.844503 13836 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33046
I20240502 14:05:34.844502 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658659912502-11301-0/minicluster-data/ts-2-root/instance:
uuid: "089dc5089de544edba8fa163340ade87"
format_stamp: "Formatted at 2024-05-02 14:05:34 on dist-test-slave-j72s"
I20240502 14:05:34.849012 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240502 14:05:34.851958 14049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:34.852675 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:05:34.852934 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658659912502-11301-0/minicluster-data/ts-2-root
uuid: "089dc5089de544edba8fa163340ade87"
format_stamp: "Formatted at 2024-05-02 14:05:34 on dist-test-slave-j72s"
I20240502 14:05:34.853188 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658659912502-11301-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658659912502-11301-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658659912502-11301-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:34.872246 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:34.873212 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:34.874506 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:34.876559 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:34.876760 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:34.876986 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:34.877130 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:34.914149 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.67:36891
I20240502 14:05:34.914242 14111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.67:36891 every 8 connection(s)
I20240502 14:05:34.918727 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:34.926534 14115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:34.927908 14116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:34.930058 14118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:34.930294 14112 heartbeater.cc:347] Connected to a master server at 127.11.9.126:38683
I20240502 14:05:34.930752 14112 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:34.931432 14112 heartbeater.cc:511] Master 127.11.9.126:38683 requested a full tablet report, sending...
I20240502 14:05:34.932065 11301 server_base.cc:1034] running on GCE node
I20240502 14:05:34.933028 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:05:34.933239 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:05:34.933409 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658734933391 us; error 0 us; skew 500 ppm
I20240502 14:05:34.933393 13836 ts_manager.cc:194] Registered new tserver with Master: 089dc5089de544edba8fa163340ade87 (127.11.9.67:36891)
I20240502 14:05:34.934166 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:34.935084 13836 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33048
I20240502 14:05:34.937099 11301 webserver.cc:460] Webserver started at http://127.11.9.68:45845/ using document root <none> and password file <none>
I20240502 14:05:34.937569 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:34.937738 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:34.937983 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:34.939085 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658659912502-11301-0/minicluster-data/ts-3-root/instance:
uuid: "63934d8091fd43ab8d789adf26e6ab07"
format_stamp: "Formatted at 2024-05-02 14:05:34 on dist-test-slave-j72s"
I20240502 14:05:34.943202 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:05:34.945993 14123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:34.946632 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:05:34.946890 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658659912502-11301-0/minicluster-data/ts-3-root
uuid: "63934d8091fd43ab8d789adf26e6ab07"
format_stamp: "Formatted at 2024-05-02 14:05:34 on dist-test-slave-j72s"
I20240502 14:05:34.947136 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658659912502-11301-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658659912502-11301-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714658659912502-11301-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:34.974144 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:34.975188 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:34.976500 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:34.978518 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:34.978704 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:34.978914 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:34.979058 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:35.016310 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.68:35667
I20240502 14:05:35.016399 14185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.68:35667 every 8 connection(s)
I20240502 14:05:35.028280 14186 heartbeater.cc:347] Connected to a master server at 127.11.9.126:38683
I20240502 14:05:35.028613 14186 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:35.029263 14186 heartbeater.cc:511] Master 127.11.9.126:38683 requested a full tablet report, sending...
I20240502 14:05:35.030963 13836 ts_manager.cc:194] Registered new tserver with Master: 63934d8091fd43ab8d789adf26e6ab07 (127.11.9.68:35667)
I20240502 14:05:35.031565 11301 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.012205994s
I20240502 14:05:35.032238 13836 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33052
I20240502 14:05:35.752972 13963 heartbeater.cc:503] Master 127.11.9.126:38683 was elected leader, sending a full tablet report...
I20240502 14:05:35.846997 14038 heartbeater.cc:503] Master 127.11.9.126:38683 was elected leader, sending a full tablet report...
I20240502 14:05:35.937363 14112 heartbeater.cc:503] Master 127.11.9.126:38683 was elected leader, sending a full tablet report...
I20240502 14:05:36.034574 14186 heartbeater.cc:503] Master 127.11.9.126:38683 was elected leader, sending a full tablet report...
I20240502 14:05:36.074123 11301 test_util.cc:268] Using random seed: 417328501
I20240502 14:05:36.094183 13836 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33066:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 14:05:36.133651 14151 tablet_service.cc:1467] Processing CreateTablet for tablet 53838ecaac33438d91a5537586f137ce (DEFAULT_TABLE table=test-workload [id=02472ab86afc4b14a6ee632c1b6ac21a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:05:36.134747 14151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53838ecaac33438d91a5537586f137ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:36.139151 14077 tablet_service.cc:1467] Processing CreateTablet for tablet 53838ecaac33438d91a5537586f137ce (DEFAULT_TABLE table=test-workload [id=02472ab86afc4b14a6ee632c1b6ac21a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:05:36.140369 14077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53838ecaac33438d91a5537586f137ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:36.141638 13928 tablet_service.cc:1467] Processing CreateTablet for tablet 53838ecaac33438d91a5537586f137ce (DEFAULT_TABLE table=test-workload [id=02472ab86afc4b14a6ee632c1b6ac21a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:05:36.142820 13928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53838ecaac33438d91a5537586f137ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:36.153396 14208 tablet_bootstrap.cc:492] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87: Bootstrap starting.
I20240502 14:05:36.155053 14207 tablet_bootstrap.cc:492] T 53838ecaac33438d91a5537586f137ce P 63934d8091fd43ab8d789adf26e6ab07: Bootstrap starting.
I20240502 14:05:36.159682 14208 tablet_bootstrap.cc:654] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:36.161270 14207 tablet_bootstrap.cc:654] T 53838ecaac33438d91a5537586f137ce P 63934d8091fd43ab8d789adf26e6ab07: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:36.161675 14209 tablet_bootstrap.cc:492] T 53838ecaac33438d91a5537586f137ce P 4b7709fd386845c5b1b11e0f80b40ee6: Bootstrap starting.
I20240502 14:05:36.165817 14208 tablet_bootstrap.cc:492] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87: No bootstrap required, opened a new log
I20240502 14:05:36.166277 14208 ts_tablet_manager.cc:1367] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20240502 14:05:36.166736 14209 tablet_bootstrap.cc:654] T 53838ecaac33438d91a5537586f137ce P 4b7709fd386845c5b1b11e0f80b40ee6: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:36.167999 14207 tablet_bootstrap.cc:492] T 53838ecaac33438d91a5537586f137ce P 63934d8091fd43ab8d789adf26e6ab07: No bootstrap required, opened a new log
I20240502 14:05:36.168502 14207 ts_tablet_manager.cc:1367] T 53838ecaac33438d91a5537586f137ce P 63934d8091fd43ab8d789adf26e6ab07: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20240502 14:05:36.168915 14208 raft_consensus.cc:348] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63934d8091fd43ab8d789adf26e6ab07" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 35667 } } peers { permanent_uuid: "089dc5089de544edba8fa163340ade87" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 36891 } } peers { permanent_uuid: "4b7709fd386845c5b1b11e0f80b40ee6" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 42985 } }
I20240502 14:05:36.169744 14208 raft_consensus.cc:374] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:36.170059 14208 raft_consensus.cc:729] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 089dc5089de544edba8fa163340ade87, State: Initialized, Role: FOLLOWER
I20240502 14:05:36.170828 14208 consensus_queue.cc:260] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63934d8091fd43ab8d789adf26e6ab07" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 35667 } } peers { permanent_uuid: "089dc5089de544edba8fa163340ade87" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 36891 } } peers { permanent_uuid: "4b7709fd386845c5b1b11e0f80b40ee6" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 42985 } }
I20240502 14:05:36.171607 14207 raft_consensus.cc:348] T 53838ecaac33438d91a5537586f137ce P 63934d8091fd43ab8d789adf26e6ab07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63934d8091fd43ab8d789adf26e6ab07" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 35667 } } peers { permanent_uuid: "089dc5089de544edba8fa163340ade87" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 36891 } } peers { permanent_uuid: "4b7709fd386845c5b1b11e0f80b40ee6" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 42985 } }
I20240502 14:05:36.172456 14207 raft_consensus.cc:374] T 53838ecaac33438d91a5537586f137ce P 63934d8091fd43ab8d789adf26e6ab07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:36.172883 14207 raft_consensus.cc:729] T 53838ecaac33438d91a5537586f137ce P 63934d8091fd43ab8d789adf26e6ab07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63934d8091fd43ab8d789adf26e6ab07, State: Initialized, Role: FOLLOWER
I20240502 14:05:36.173449 14209 tablet_bootstrap.cc:492] T 53838ecaac33438d91a5537586f137ce P 4b7709fd386845c5b1b11e0f80b40ee6: No bootstrap required, opened a new log
I20240502 14:05:36.174002 14208 ts_tablet_manager.cc:1398] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240502 14:05:36.174575 14209 ts_tablet_manager.cc:1367] T 53838ecaac33438d91a5537586f137ce P 4b7709fd386845c5b1b11e0f80b40ee6: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20240502 14:05:36.174651 14207 consensus_queue.cc:260] T 53838ecaac33438d91a5537586f137ce P 63934d8091fd43ab8d789adf26e6ab07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63934d8091fd43ab8d789adf26e6ab07" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 35667 } } peers { permanent_uuid: "089dc5089de544edba8fa163340ade87" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 36891 } } peers { permanent_uuid: "4b7709fd386845c5b1b11e0f80b40ee6" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 42985 } }
I20240502 14:05:36.177237 14209 raft_consensus.cc:348] T 53838ecaac33438d91a5537586f137ce P 4b7709fd386845c5b1b11e0f80b40ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63934d8091fd43ab8d789adf26e6ab07" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 35667 } } peers { permanent_uuid: "089dc5089de544edba8fa163340ade87" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 36891 } } peers { permanent_uuid: "4b7709fd386845c5b1b11e0f80b40ee6" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 42985 } }
I20240502 14:05:36.177740 14207 ts_tablet_manager.cc:1398] T 53838ecaac33438d91a5537586f137ce P 63934d8091fd43ab8d789adf26e6ab07: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20240502 14:05:36.178020 14209 raft_consensus.cc:374] T 53838ecaac33438d91a5537586f137ce P 4b7709fd386845c5b1b11e0f80b40ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:36.178407 14209 raft_consensus.cc:729] T 53838ecaac33438d91a5537586f137ce P 4b7709fd386845c5b1b11e0f80b40ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7709fd386845c5b1b11e0f80b40ee6, State: Initialized, Role: FOLLOWER
I20240502 14:05:36.179140 14209 consensus_queue.cc:260] T 53838ecaac33438d91a5537586f137ce P 4b7709fd386845c5b1b11e0f80b40ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63934d8091fd43ab8d789adf26e6ab07" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 35667 } } peers { permanent_uuid: "089dc5089de544edba8fa163340ade87" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 36891 } } peers { permanent_uuid: "4b7709fd386845c5b1b11e0f80b40ee6" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 42985 } }
I20240502 14:05:36.182852 14209 ts_tablet_manager.cc:1398] T 53838ecaac33438d91a5537586f137ce P 4b7709fd386845c5b1b11e0f80b40ee6: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20240502 14:05:36.204157 14213 raft_consensus.cc:482] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:36.204536 14213 raft_consensus.cc:504] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63934d8091fd43ab8d789adf26e6ab07" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 35667 } } peers { permanent_uuid: "089dc5089de544edba8fa163340ade87" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 36891 } } peers { permanent_uuid: "4b7709fd386845c5b1b11e0f80b40ee6" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 42985 } }
I20240502 14:05:36.206161 14213 leader_election.cc:288] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63934d8091fd43ab8d789adf26e6ab07 (127.11.9.68:35667), 4b7709fd386845c5b1b11e0f80b40ee6 (127.11.9.65:42985)
I20240502 14:05:36.215974 14161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53838ecaac33438d91a5537586f137ce" candidate_uuid: "089dc5089de544edba8fa163340ade87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63934d8091fd43ab8d789adf26e6ab07" is_pre_election: true
I20240502 14:05:36.216804 14161 raft_consensus.cc:2454] T 53838ecaac33438d91a5537586f137ce P 63934d8091fd43ab8d789adf26e6ab07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 089dc5089de544edba8fa163340ade87 in term 0.
I20240502 14:05:36.217257 13938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53838ecaac33438d91a5537586f137ce" candidate_uuid: "089dc5089de544edba8fa163340ade87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b7709fd386845c5b1b11e0f80b40ee6" is_pre_election: true
I20240502 14:05:36.217901 13938 raft_consensus.cc:2454] T 53838ecaac33438d91a5537586f137ce P 4b7709fd386845c5b1b11e0f80b40ee6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 089dc5089de544edba8fa163340ade87 in term 0.
I20240502 14:05:36.217886 14052 leader_election.cc:302] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 089dc5089de544edba8fa163340ade87, 63934d8091fd43ab8d789adf26e6ab07; no voters:
I20240502 14:05:36.219108 14213 raft_consensus.cc:2781] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:36.219372 14213 raft_consensus.cc:482] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:36.219607 14213 raft_consensus.cc:3037] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:36.223927 14213 raft_consensus.cc:504] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63934d8091fd43ab8d789adf26e6ab07" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 35667 } } peers { permanent_uuid: "089dc5089de544edba8fa163340ade87" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 36891 } } peers { permanent_uuid: "4b7709fd386845c5b1b11e0f80b40ee6" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 42985 } }
I20240502 14:05:36.225231 14213 leader_election.cc:288] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87 [CANDIDATE]: Term 1 election: Requested vote from peers 63934d8091fd43ab8d789adf26e6ab07 (127.11.9.68:35667), 4b7709fd386845c5b1b11e0f80b40ee6 (127.11.9.65:42985)
I20240502 14:05:36.226003 14161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53838ecaac33438d91a5537586f137ce" candidate_uuid: "089dc5089de544edba8fa163340ade87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63934d8091fd43ab8d789adf26e6ab07"
I20240502 14:05:36.226214 13938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53838ecaac33438d91a5537586f137ce" candidate_uuid: "089dc5089de544edba8fa163340ade87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b7709fd386845c5b1b11e0f80b40ee6"
I20240502 14:05:36.226555 14161 raft_consensus.cc:3037] T 53838ecaac33438d91a5537586f137ce P 63934d8091fd43ab8d789adf26e6ab07 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:36.226686 13938 raft_consensus.cc:3037] T 53838ecaac33438d91a5537586f137ce P 4b7709fd386845c5b1b11e0f80b40ee6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:36.233232 13938 raft_consensus.cc:2454] T 53838ecaac33438d91a5537586f137ce P 4b7709fd386845c5b1b11e0f80b40ee6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 089dc5089de544edba8fa163340ade87 in term 1.
I20240502 14:05:36.233230 14161 raft_consensus.cc:2454] T 53838ecaac33438d91a5537586f137ce P 63934d8091fd43ab8d789adf26e6ab07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 089dc5089de544edba8fa163340ade87 in term 1.
W20240502 14:05:36.233645 13964 tablet.cc:2369] T 53838ecaac33438d91a5537586f137ce P 4b7709fd386845c5b1b11e0f80b40ee6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:05:36.234440 14052 leader_election.cc:302] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 089dc5089de544edba8fa163340ade87, 4b7709fd386845c5b1b11e0f80b40ee6; no voters:
I20240502 14:05:36.235131 14213 raft_consensus.cc:2781] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:36.236162 14213 raft_consensus.cc:686] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87 [term 1 LEADER]: Becoming Leader. State: Replica: 089dc5089de544edba8fa163340ade87, State: Running, Role: LEADER
I20240502 14:05:36.236763 14213 consensus_queue.cc:237] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63934d8091fd43ab8d789adf26e6ab07" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 35667 } } peers { permanent_uuid: "089dc5089de544edba8fa163340ade87" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 36891 } } peers { permanent_uuid: "4b7709fd386845c5b1b11e0f80b40ee6" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 42985 } }
I20240502 14:05:36.243863 13835 catalog_manager.cc:5484] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 089dc5089de544edba8fa163340ade87 (127.11.9.67). New cstate: current_term: 1 leader_uuid: "089dc5089de544edba8fa163340ade87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63934d8091fd43ab8d789adf26e6ab07" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 35667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "089dc5089de544edba8fa163340ade87" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 36891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b7709fd386845c5b1b11e0f80b40ee6" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 42985 } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:36.615198 13889 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:36.616490 13889 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:36.616812 13889 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:36.641222 14213 consensus_queue.cc:1035] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63934d8091fd43ab8d789adf26e6ab07" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 35667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:36.651794 14218 consensus_queue.cc:1035] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7709fd386845c5b1b11e0f80b40ee6" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 42985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:05:38.617844 13889 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:38.619144 13889 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:38.619488 13889 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:40.620780 13889 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:40.622506 13889 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:40.622792 13889 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 14:05:41.833271 14034 debug-util.cc:398] Leaking SignalData structure 0x7b0800119980 after lost signal to thread 11304
W20240502 14:05:41.834123 14034 debug-util.cc:398] Leaking SignalData structure 0x7b0800233d20 after lost signal to thread 13870
W20240502 14:05:41.834978 14034 debug-util.cc:398] Leaking SignalData structure 0x7b0800174d80 after lost signal to thread 13962
W20240502 14:05:41.835768 14034 debug-util.cc:398] Leaking SignalData structure 0x7b08001594c0 after lost signal to thread 14037
W20240502 14:05:41.836649 14034 debug-util.cc:398] Leaking SignalData structure 0x7b080023dca0 after lost signal to thread 14111
W20240502 14:05:41.837422 14034 debug-util.cc:398] Leaking SignalData structure 0x7b0800049f80 after lost signal to thread 14185
W20240502 14:05:42.608078 14108 debug-util.cc:398] Leaking SignalData structure 0x7b08000f4640 after lost signal to thread 11304
W20240502 14:05:42.608865 14108 debug-util.cc:398] Leaking SignalData structure 0x7b08000cdb80 after lost signal to thread 13870
W20240502 14:05:42.609809 14108 debug-util.cc:398] Leaking SignalData structure 0x7b08001b2940 after lost signal to thread 13962
W20240502 14:05:42.610574 14108 debug-util.cc:398] Leaking SignalData structure 0x7b080000a360 after lost signal to thread 14037
W20240502 14:05:42.611356 14108 debug-util.cc:398] Leaking SignalData structure 0x7b080015c080 after lost signal to thread 14111
W20240502 14:05:42.612133 14108 debug-util.cc:398] Leaking SignalData structure 0x7b080015c0c0 after lost signal to thread 14185
I20240502 14:05:42.623829 13889 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:42.625634 13889 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:42.626019 13889 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:44.626900 13889 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:44.628131 13889 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:44.628507 13889 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:46.629240 13889 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:46.630476 13889 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:46.630755 13889 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:46.830732 11301 tablet_server.cc:178] TabletServer@127.11.9.65:0 shutting down...
I20240502 14:05:46.853588 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:05:46.854212 11301 tablet_replica.cc:330] T 53838ecaac33438d91a5537586f137ce P 4b7709fd386845c5b1b11e0f80b40ee6: stopping tablet replica
I20240502 14:05:46.854733 11301 raft_consensus.cc:2229] T 53838ecaac33438d91a5537586f137ce P 4b7709fd386845c5b1b11e0f80b40ee6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:46.855316 11301 raft_consensus.cc:2258] T 53838ecaac33438d91a5537586f137ce P 4b7709fd386845c5b1b11e0f80b40ee6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:46.874903 11301 tablet_server.cc:194] TabletServer@127.11.9.65:0 shutdown complete.
I20240502 14:05:46.886538 11301 tablet_server.cc:178] TabletServer@127.11.9.66:0 shutting down...
I20240502 14:05:46.909651 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:05:46.942482 11301 tablet_server.cc:194] TabletServer@127.11.9.66:0 shutdown complete.
I20240502 14:05:46.951192 11301 tablet_server.cc:178] TabletServer@127.11.9.67:0 shutting down...
I20240502 14:05:46.969604 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:05:46.970238 11301 tablet_replica.cc:330] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87: stopping tablet replica
I20240502 14:05:46.970765 11301 raft_consensus.cc:2229] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:05:46.971727 11301 raft_consensus.cc:2258] T 53838ecaac33438d91a5537586f137ce P 089dc5089de544edba8fa163340ade87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:47.007467 11301 tablet_server.cc:194] TabletServer@127.11.9.67:0 shutdown complete.
I20240502 14:05:47.016286 11301 tablet_server.cc:178] TabletServer@127.11.9.68:0 shutting down...
I20240502 14:05:47.035224 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:05:47.035879 11301 tablet_replica.cc:330] T 53838ecaac33438d91a5537586f137ce P 63934d8091fd43ab8d789adf26e6ab07: stopping tablet replica
I20240502 14:05:47.036458 11301 raft_consensus.cc:2229] T 53838ecaac33438d91a5537586f137ce P 63934d8091fd43ab8d789adf26e6ab07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:47.036893 11301 raft_consensus.cc:2258] T 53838ecaac33438d91a5537586f137ce P 63934d8091fd43ab8d789adf26e6ab07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:47.055509 11301 tablet_server.cc:194] TabletServer@127.11.9.68:0 shutdown complete.
I20240502 14:05:47.064400 11301 master.cc:537] Master@127.11.9.126:38683 shutting down...
I20240502 14:05:47.080633 11301 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8258f89f0a6b400a82af70c742262a67 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:05:47.081187 11301 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8258f89f0a6b400a82af70c742262a67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:47.081490 11301 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8258f89f0a6b400a82af70c742262a67: stopping tablet replica
I20240502 14:05:47.099054 11301 master.cc:552] Master@127.11.9.126:38683 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12645 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240502 14:05:47.135241 11301 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.9.126:40313
I20240502 14:05:47.136257 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:47.142395 14230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:47.143234 14231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:47.143898 14233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:47.144686 11301 server_base.cc:1034] running on GCE node
I20240502 14:05:47.145478 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:05:47.145709 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:05:47.145857 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658747145839 us; error 0 us; skew 500 ppm
I20240502 14:05:47.146315 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:47.148553 11301 webserver.cc:460] Webserver started at http://127.11.9.126:34383/ using document root <none> and password file <none>
I20240502 14:05:47.148999 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:47.149161 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:47.149421 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:47.150578 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658659912502-11301-0/minicluster-data/master-0-root/instance:
uuid: "ea9f0c56eb144f4fa405d3a86b75e4cd"
format_stamp: "Formatted at 2024-05-02 14:05:47 on dist-test-slave-j72s"
I20240502 14:05:47.154789 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20240502 14:05:47.157866 14238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:47.158500 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240502 14:05:47.158744 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658659912502-11301-0/minicluster-data/master-0-root
uuid: "ea9f0c56eb144f4fa405d3a86b75e4cd"
format_stamp: "Formatted at 2024-05-02 14:05:47 on dist-test-slave-j72s"
I20240502 14:05:47.158975 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658659912502-11301-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658659912502-11301-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658659912502-11301-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:47.176690 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:47.177765 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:47.211671 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.126:40313
I20240502 14:05:47.211767 14289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.126:40313 every 8 connection(s)
I20240502 14:05:47.215288 14290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:47.225015 14290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9f0c56eb144f4fa405d3a86b75e4cd: Bootstrap starting.
I20240502 14:05:47.228921 14290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea9f0c56eb144f4fa405d3a86b75e4cd: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:47.232640 14290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9f0c56eb144f4fa405d3a86b75e4cd: No bootstrap required, opened a new log
I20240502 14:05:47.234442 14290 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea9f0c56eb144f4fa405d3a86b75e4cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9f0c56eb144f4fa405d3a86b75e4cd" member_type: VOTER }
I20240502 14:05:47.234853 14290 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea9f0c56eb144f4fa405d3a86b75e4cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:47.235008 14290 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea9f0c56eb144f4fa405d3a86b75e4cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9f0c56eb144f4fa405d3a86b75e4cd, State: Initialized, Role: FOLLOWER
I20240502 14:05:47.235450 14290 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea9f0c56eb144f4fa405d3a86b75e4cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9f0c56eb144f4fa405d3a86b75e4cd" member_type: VOTER }
I20240502 14:05:47.235832 14290 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea9f0c56eb144f4fa405d3a86b75e4cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:05:47.236017 14290 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea9f0c56eb144f4fa405d3a86b75e4cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:05:47.236212 14290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea9f0c56eb144f4fa405d3a86b75e4cd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:47.239640 14290 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea9f0c56eb144f4fa405d3a86b75e4cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9f0c56eb144f4fa405d3a86b75e4cd" member_type: VOTER }
I20240502 14:05:47.240020 14290 leader_election.cc:302] T 00000000000000000000000000000000 P ea9f0c56eb144f4fa405d3a86b75e4cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea9f0c56eb144f4fa405d3a86b75e4cd; no voters:
I20240502 14:05:47.240931 14290 leader_election.cc:288] T 00000000000000000000000000000000 P ea9f0c56eb144f4fa405d3a86b75e4cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:05:47.241283 14293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea9f0c56eb144f4fa405d3a86b75e4cd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:47.242657 14293 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea9f0c56eb144f4fa405d3a86b75e4cd [term 1 LEADER]: Becoming Leader. State: Replica: ea9f0c56eb144f4fa405d3a86b75e4cd, State: Running, Role: LEADER
I20240502 14:05:47.243140 14293 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea9f0c56eb144f4fa405d3a86b75e4cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9f0c56eb144f4fa405d3a86b75e4cd" member_type: VOTER }
I20240502 14:05:47.243628 14290 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea9f0c56eb144f4fa405d3a86b75e4cd [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:05:47.247620 14294 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9f0c56eb144f4fa405d3a86b75e4cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea9f0c56eb144f4fa405d3a86b75e4cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9f0c56eb144f4fa405d3a86b75e4cd" member_type: VOTER } }
I20240502 14:05:47.248689 14295 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9f0c56eb144f4fa405d3a86b75e4cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea9f0c56eb144f4fa405d3a86b75e4cd. Latest consensus state: current_term: 1 leader_uuid: "ea9f0c56eb144f4fa405d3a86b75e4cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9f0c56eb144f4fa405d3a86b75e4cd" member_type: VOTER } }
I20240502 14:05:47.249240 14294 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9f0c56eb144f4fa405d3a86b75e4cd [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:47.249293 14295 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9f0c56eb144f4fa405d3a86b75e4cd [sys.catalog]: This master's current role is: LEADER
I20240502 14:05:47.251202 14302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:05:47.256888 14302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:05:47.257788 11301 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:05:47.265084 14302 catalog_manager.cc:1347] Generated new cluster ID: df24f9d25b26461c919de5ed3a24c0ae
I20240502 14:05:47.265326 14302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:05:47.301273 14302 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:05:47.302605 14302 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:05:47.313534 14302 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ea9f0c56eb144f4fa405d3a86b75e4cd: Generated new TSK 0
I20240502 14:05:47.314049 14302 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:05:47.324153 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:47.330354 14311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:47.331642 14312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:47.332336 14314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:47.333153 11301 server_base.cc:1034] running on GCE node
I20240502 14:05:47.333948 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:05:47.334123 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:05:47.334242 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658747334231 us; error 0 us; skew 500 ppm
I20240502 14:05:47.334677 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:47.336735 11301 webserver.cc:460] Webserver started at http://127.11.9.65:42027/ using document root <none> and password file <none>
I20240502 14:05:47.337128 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:47.337268 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:47.337461 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:47.338477 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658659912502-11301-0/minicluster-data/ts-0-root/instance:
uuid: "cfd772dccab34b3ab0801ab5b4ccebc3"
format_stamp: "Formatted at 2024-05-02 14:05:47 on dist-test-slave-j72s"
I20240502 14:05:47.342528 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:05:47.345388 14319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:47.346083 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:05:47.346310 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658659912502-11301-0/minicluster-data/ts-0-root
uuid: "cfd772dccab34b3ab0801ab5b4ccebc3"
format_stamp: "Formatted at 2024-05-02 14:05:47 on dist-test-slave-j72s"
I20240502 14:05:47.346531 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658659912502-11301-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658659912502-11301-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658659912502-11301-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:47.388607 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:47.389700 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:47.391073 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:47.393329 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:47.393487 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:47.393728 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:47.393882 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:47.431362 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.65:38773
I20240502 14:05:47.431447 14381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.65:38773 every 8 connection(s)
I20240502 14:05:47.435999 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:47.444711 14387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:47.444808 14386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:47.446794 11301 server_base.cc:1034] running on GCE node
W20240502 14:05:47.447775 14389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:47.448796 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:05:47.449065 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:05:47.449235 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658747449215 us; error 0 us; skew 500 ppm
I20240502 14:05:47.449447 14382 heartbeater.cc:347] Connected to a master server at 127.11.9.126:40313
I20240502 14:05:47.449836 14382 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:47.449925 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:47.450850 14382 heartbeater.cc:511] Master 127.11.9.126:40313 requested a full tablet report, sending...
I20240502 14:05:47.452739 11301 webserver.cc:460] Webserver started at http://127.11.9.66:33261/ using document root <none> and password file <none>
I20240502 14:05:47.453222 14255 ts_manager.cc:194] Registered new tserver with Master: cfd772dccab34b3ab0801ab5b4ccebc3 (127.11.9.65:38773)
I20240502 14:05:47.453377 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:47.453677 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:47.454005 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:47.455441 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658659912502-11301-0/minicluster-data/ts-1-root/instance:
uuid: "fc188f6a27d448a4847a7601c7c3ac55"
format_stamp: "Formatted at 2024-05-02 14:05:47 on dist-test-slave-j72s"
I20240502 14:05:47.455695 14255 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57676
I20240502 14:05:47.460449 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 14:05:47.463701 14394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:47.464391 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:05:47.464617 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658659912502-11301-0/minicluster-data/ts-1-root
uuid: "fc188f6a27d448a4847a7601c7c3ac55"
format_stamp: "Formatted at 2024-05-02 14:05:47 on dist-test-slave-j72s"
I20240502 14:05:47.464857 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658659912502-11301-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658659912502-11301-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658659912502-11301-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:47.473798 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:47.474784 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:47.475984 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:47.478035 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:47.478200 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:47.478425 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:47.478559 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:47.515504 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.66:34383
I20240502 14:05:47.515591 14456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.66:34383 every 8 connection(s)
I20240502 14:05:47.520104 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:05:47.527516 14460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:05:47.529280 14461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:47.530776 11301 server_base.cc:1034] running on GCE node
W20240502 14:05:47.531746 14463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:05:47.532369 14457 heartbeater.cc:347] Connected to a master server at 127.11.9.126:40313
I20240502 14:05:47.532616 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 14:05:47.532682 14457 heartbeater.cc:464] Registering TS with master...
W20240502 14:05:47.532876 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:05:47.533083 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658747533068 us; error 0 us; skew 500 ppm
I20240502 14:05:47.533562 14457 heartbeater.cc:511] Master 127.11.9.126:40313 requested a full tablet report, sending...
I20240502 14:05:47.533666 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:05:47.535600 14255 ts_manager.cc:194] Registered new tserver with Master: fc188f6a27d448a4847a7601c7c3ac55 (127.11.9.66:34383)
I20240502 14:05:47.536514 11301 webserver.cc:460] Webserver started at http://127.11.9.67:35623/ using document root <none> and password file <none>
I20240502 14:05:47.537112 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:05:47.537377 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:05:47.537446 14255 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57684
I20240502 14:05:47.537775 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:05:47.539026 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658659912502-11301-0/minicluster-data/ts-2-root/instance:
uuid: "5dc04d5dffb14379bdec724eedf369c3"
format_stamp: "Formatted at 2024-05-02 14:05:47 on dist-test-slave-j72s"
I20240502 14:05:47.543589 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:05:47.546661 14475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:47.547463 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:05:47.547717 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658659912502-11301-0/minicluster-data/ts-2-root
uuid: "5dc04d5dffb14379bdec724eedf369c3"
format_stamp: "Formatted at 2024-05-02 14:05:47 on dist-test-slave-j72s"
I20240502 14:05:47.547981 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658659912502-11301-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658659912502-11301-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714658659912502-11301-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:05:47.569175 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:05:47.570210 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:05:47.571568 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:05:47.573810 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:05:47.574007 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:47.574218 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:05:47.574373 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:05:47.611112 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.67:34373
I20240502 14:05:47.611209 14537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.67:34373 every 8 connection(s)
I20240502 14:05:47.624152 14538 heartbeater.cc:347] Connected to a master server at 127.11.9.126:40313
I20240502 14:05:47.624500 14538 heartbeater.cc:464] Registering TS with master...
I20240502 14:05:47.625135 14538 heartbeater.cc:511] Master 127.11.9.126:40313 requested a full tablet report, sending...
I20240502 14:05:47.626793 14255 ts_manager.cc:194] Registered new tserver with Master: 5dc04d5dffb14379bdec724eedf369c3 (127.11.9.67:34373)
I20240502 14:05:47.627306 11301 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013015106s
I20240502 14:05:47.628116 14255 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57700
I20240502 14:05:48.458642 14382 heartbeater.cc:503] Master 127.11.9.126:40313 was elected leader, sending a full tablet report...
I20240502 14:05:48.539844 14457 heartbeater.cc:503] Master 127.11.9.126:40313 was elected leader, sending a full tablet report...
I20240502 14:05:48.630456 14538 heartbeater.cc:503] Master 127.11.9.126:40313 was elected leader, sending a full tablet report...
I20240502 14:05:48.669273 11301 test_util.cc:268] Using random seed: 429923652
I20240502 14:05:48.688958 14255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57718:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:05:48.690786 14255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:05:48.730010 14503 tablet_service.cc:1467] Processing CreateTablet for tablet d6c164a264eb4630912a74d65efb7387 (DEFAULT_TABLE table=test-workload [id=b8c48cfe732a4698b4bfb29a080b7a11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:05:48.731000 14503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c164a264eb4630912a74d65efb7387. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:48.734897 14422 tablet_service.cc:1467] Processing CreateTablet for tablet d6c164a264eb4630912a74d65efb7387 (DEFAULT_TABLE table=test-workload [id=b8c48cfe732a4698b4bfb29a080b7a11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:05:48.735165 14347 tablet_service.cc:1467] Processing CreateTablet for tablet d6c164a264eb4630912a74d65efb7387 (DEFAULT_TABLE table=test-workload [id=b8c48cfe732a4698b4bfb29a080b7a11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:05:48.735987 14422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c164a264eb4630912a74d65efb7387. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:48.736285 14347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c164a264eb4630912a74d65efb7387. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:48.748494 14559 tablet_bootstrap.cc:492] T d6c164a264eb4630912a74d65efb7387 P 5dc04d5dffb14379bdec724eedf369c3: Bootstrap starting.
I20240502 14:05:48.748898 14560 tablet_bootstrap.cc:492] T d6c164a264eb4630912a74d65efb7387 P cfd772dccab34b3ab0801ab5b4ccebc3: Bootstrap starting.
I20240502 14:05:48.752589 14561 tablet_bootstrap.cc:492] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55: Bootstrap starting.
I20240502 14:05:48.754709 14560 tablet_bootstrap.cc:654] T d6c164a264eb4630912a74d65efb7387 P cfd772dccab34b3ab0801ab5b4ccebc3: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:48.756255 14559 tablet_bootstrap.cc:654] T d6c164a264eb4630912a74d65efb7387 P 5dc04d5dffb14379bdec724eedf369c3: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:48.758785 14561 tablet_bootstrap.cc:654] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55: Neither blocks nor log segments found. Creating new log.
I20240502 14:05:48.758957 14560 tablet_bootstrap.cc:492] T d6c164a264eb4630912a74d65efb7387 P cfd772dccab34b3ab0801ab5b4ccebc3: No bootstrap required, opened a new log
I20240502 14:05:48.759516 14560 ts_tablet_manager.cc:1367] T d6c164a264eb4630912a74d65efb7387 P cfd772dccab34b3ab0801ab5b4ccebc3: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240502 14:05:48.762936 14560 raft_consensus.cc:348] T d6c164a264eb4630912a74d65efb7387 P cfd772dccab34b3ab0801ab5b4ccebc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc04d5dffb14379bdec724eedf369c3" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34373 } } peers { permanent_uuid: "cfd772dccab34b3ab0801ab5b4ccebc3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 38773 } } peers { permanent_uuid: "fc188f6a27d448a4847a7601c7c3ac55" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 34383 } }
I20240502 14:05:48.763823 14560 raft_consensus.cc:374] T d6c164a264eb4630912a74d65efb7387 P cfd772dccab34b3ab0801ab5b4ccebc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:48.764158 14560 raft_consensus.cc:729] T d6c164a264eb4630912a74d65efb7387 P cfd772dccab34b3ab0801ab5b4ccebc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfd772dccab34b3ab0801ab5b4ccebc3, State: Initialized, Role: FOLLOWER
I20240502 14:05:48.764849 14559 tablet_bootstrap.cc:492] T d6c164a264eb4630912a74d65efb7387 P 5dc04d5dffb14379bdec724eedf369c3: No bootstrap required, opened a new log
I20240502 14:05:48.764937 14560 consensus_queue.cc:260] T d6c164a264eb4630912a74d65efb7387 P cfd772dccab34b3ab0801ab5b4ccebc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc04d5dffb14379bdec724eedf369c3" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34373 } } peers { permanent_uuid: "cfd772dccab34b3ab0801ab5b4ccebc3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 38773 } } peers { permanent_uuid: "fc188f6a27d448a4847a7601c7c3ac55" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 34383 } }
I20240502 14:05:48.765444 14559 ts_tablet_manager.cc:1367] T d6c164a264eb4630912a74d65efb7387 P 5dc04d5dffb14379bdec724eedf369c3: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.004s
I20240502 14:05:48.765951 14561 tablet_bootstrap.cc:492] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55: No bootstrap required, opened a new log
I20240502 14:05:48.766394 14561 ts_tablet_manager.cc:1367] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.005s
I20240502 14:05:48.768226 14559 raft_consensus.cc:348] T d6c164a264eb4630912a74d65efb7387 P 5dc04d5dffb14379bdec724eedf369c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc04d5dffb14379bdec724eedf369c3" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34373 } } peers { permanent_uuid: "cfd772dccab34b3ab0801ab5b4ccebc3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 38773 } } peers { permanent_uuid: "fc188f6a27d448a4847a7601c7c3ac55" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 34383 } }
I20240502 14:05:48.769059 14560 ts_tablet_manager.cc:1398] T d6c164a264eb4630912a74d65efb7387 P cfd772dccab34b3ab0801ab5b4ccebc3: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240502 14:05:48.768944 14561 raft_consensus.cc:348] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc04d5dffb14379bdec724eedf369c3" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34373 } } peers { permanent_uuid: "cfd772dccab34b3ab0801ab5b4ccebc3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 38773 } } peers { permanent_uuid: "fc188f6a27d448a4847a7601c7c3ac55" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 34383 } }
I20240502 14:05:48.769621 14559 raft_consensus.cc:374] T d6c164a264eb4630912a74d65efb7387 P 5dc04d5dffb14379bdec724eedf369c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:48.769726 14561 raft_consensus.cc:374] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:05:48.769999 14559 raft_consensus.cc:729] T d6c164a264eb4630912a74d65efb7387 P 5dc04d5dffb14379bdec724eedf369c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dc04d5dffb14379bdec724eedf369c3, State: Initialized, Role: FOLLOWER
I20240502 14:05:48.770174 14561 raft_consensus.cc:729] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc188f6a27d448a4847a7601c7c3ac55, State: Initialized, Role: FOLLOWER
I20240502 14:05:48.770697 14559 consensus_queue.cc:260] T d6c164a264eb4630912a74d65efb7387 P 5dc04d5dffb14379bdec724eedf369c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc04d5dffb14379bdec724eedf369c3" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34373 } } peers { permanent_uuid: "cfd772dccab34b3ab0801ab5b4ccebc3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 38773 } } peers { permanent_uuid: "fc188f6a27d448a4847a7601c7c3ac55" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 34383 } }
I20240502 14:05:48.771091 14561 consensus_queue.cc:260] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc04d5dffb14379bdec724eedf369c3" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34373 } } peers { permanent_uuid: "cfd772dccab34b3ab0801ab5b4ccebc3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 38773 } } peers { permanent_uuid: "fc188f6a27d448a4847a7601c7c3ac55" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 34383 } }
I20240502 14:05:48.775616 14567 raft_consensus.cc:482] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:48.776238 14567 raft_consensus.cc:504] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc04d5dffb14379bdec724eedf369c3" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34373 } } peers { permanent_uuid: "cfd772dccab34b3ab0801ab5b4ccebc3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 38773 } } peers { permanent_uuid: "fc188f6a27d448a4847a7601c7c3ac55" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 34383 } }
I20240502 14:05:48.780217 14561 ts_tablet_manager.cc:1398] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55: Time spent starting tablet: real 0.013s user 0.004s sys 0.002s
I20240502 14:05:48.786365 14567 leader_election.cc:288] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dc04d5dffb14379bdec724eedf369c3 (127.11.9.67:34373), cfd772dccab34b3ab0801ab5b4ccebc3 (127.11.9.65:38773)
I20240502 14:05:48.798939 14559 ts_tablet_manager.cc:1398] T d6c164a264eb4630912a74d65efb7387 P 5dc04d5dffb14379bdec724eedf369c3: Time spent starting tablet: real 0.033s user 0.008s sys 0.023s
I20240502 14:05:48.802462 14513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c164a264eb4630912a74d65efb7387" candidate_uuid: "fc188f6a27d448a4847a7601c7c3ac55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc04d5dffb14379bdec724eedf369c3" is_pre_election: true
I20240502 14:05:48.803332 14513 raft_consensus.cc:2454] T d6c164a264eb4630912a74d65efb7387 P 5dc04d5dffb14379bdec724eedf369c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc188f6a27d448a4847a7601c7c3ac55 in term 0.
I20240502 14:05:48.804581 14397 leader_election.cc:302] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dc04d5dffb14379bdec724eedf369c3, fc188f6a27d448a4847a7601c7c3ac55; no voters:
I20240502 14:05:48.805413 14567 raft_consensus.cc:2781] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:05:48.805763 14567 raft_consensus.cc:482] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:05:48.806105 14567 raft_consensus.cc:3037] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:48.806810 14357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c164a264eb4630912a74d65efb7387" candidate_uuid: "fc188f6a27d448a4847a7601c7c3ac55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfd772dccab34b3ab0801ab5b4ccebc3" is_pre_election: true
I20240502 14:05:48.807493 14357 raft_consensus.cc:2454] T d6c164a264eb4630912a74d65efb7387 P cfd772dccab34b3ab0801ab5b4ccebc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc188f6a27d448a4847a7601c7c3ac55 in term 0.
I20240502 14:05:48.811772 14567 raft_consensus.cc:504] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc04d5dffb14379bdec724eedf369c3" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34373 } } peers { permanent_uuid: "cfd772dccab34b3ab0801ab5b4ccebc3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 38773 } } peers { permanent_uuid: "fc188f6a27d448a4847a7601c7c3ac55" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 34383 } }
I20240502 14:05:48.813289 14567 leader_election.cc:288] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55 [CANDIDATE]: Term 1 election: Requested vote from peers 5dc04d5dffb14379bdec724eedf369c3 (127.11.9.67:34373), cfd772dccab34b3ab0801ab5b4ccebc3 (127.11.9.65:38773)
I20240502 14:05:48.813922 14513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c164a264eb4630912a74d65efb7387" candidate_uuid: "fc188f6a27d448a4847a7601c7c3ac55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc04d5dffb14379bdec724eedf369c3"
I20240502 14:05:48.814206 14357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c164a264eb4630912a74d65efb7387" candidate_uuid: "fc188f6a27d448a4847a7601c7c3ac55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfd772dccab34b3ab0801ab5b4ccebc3"
I20240502 14:05:48.814427 14513 raft_consensus.cc:3037] T d6c164a264eb4630912a74d65efb7387 P 5dc04d5dffb14379bdec724eedf369c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:48.814772 14357 raft_consensus.cc:3037] T d6c164a264eb4630912a74d65efb7387 P cfd772dccab34b3ab0801ab5b4ccebc3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:05:48.820046 14513 raft_consensus.cc:2454] T d6c164a264eb4630912a74d65efb7387 P 5dc04d5dffb14379bdec724eedf369c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc188f6a27d448a4847a7601c7c3ac55 in term 1.
I20240502 14:05:48.820102 14357 raft_consensus.cc:2454] T d6c164a264eb4630912a74d65efb7387 P cfd772dccab34b3ab0801ab5b4ccebc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc188f6a27d448a4847a7601c7c3ac55 in term 1.
I20240502 14:05:48.821018 14397 leader_election.cc:302] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dc04d5dffb14379bdec724eedf369c3, fc188f6a27d448a4847a7601c7c3ac55; no voters:
I20240502 14:05:48.821640 14567 raft_consensus.cc:2781] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:05:48.821997 14567 raft_consensus.cc:686] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55 [term 1 LEADER]: Becoming Leader. State: Replica: fc188f6a27d448a4847a7601c7c3ac55, State: Running, Role: LEADER
I20240502 14:05:48.822549 14567 consensus_queue.cc:237] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc04d5dffb14379bdec724eedf369c3" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34373 } } peers { permanent_uuid: "cfd772dccab34b3ab0801ab5b4ccebc3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 38773 } } peers { permanent_uuid: "fc188f6a27d448a4847a7601c7c3ac55" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 34383 } }
I20240502 14:05:48.829872 14255 catalog_manager.cc:5484] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc188f6a27d448a4847a7601c7c3ac55 (127.11.9.66). New cstate: current_term: 1 leader_uuid: "fc188f6a27d448a4847a7601c7c3ac55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc04d5dffb14379bdec724eedf369c3" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfd772dccab34b3ab0801ab5b4ccebc3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 38773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc188f6a27d448a4847a7601c7c3ac55" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 34383 } health_report { overall_health: HEALTHY } } }
I20240502 14:05:49.255185 14303 placement_policy_util.cc:407] tablet d6c164a264eb4630912a74d65efb7387: detected majority of replicas (2 of 3) at location L0
I20240502 14:05:49.256704 14308 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:49.258131 14308 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:49.258569 14308 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:49.413337 14572 consensus_queue.cc:1035] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dc04d5dffb14379bdec724eedf369c3" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 34373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:49.426592 14577 consensus_queue.cc:1035] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfd772dccab34b3ab0801ab5b4ccebc3" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 38773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:05:50.263648 14303 placement_policy_util.cc:407] tablet d6c164a264eb4630912a74d65efb7387: detected majority of replicas (2 of 3) at location L0
I20240502 14:05:51.259745 14308 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:51.261533 14308 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:51.261844 14308 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:51.271648 14303 placement_policy_util.cc:407] tablet d6c164a264eb4630912a74d65efb7387: detected majority of replicas (2 of 3) at location L0
I20240502 14:05:52.279388 14303 placement_policy_util.cc:407] tablet d6c164a264eb4630912a74d65efb7387: detected majority of replicas (2 of 3) at location L0
I20240502 14:05:53.262707 14308 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:53.263988 14308 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:53.264273 14308 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:53.286859 14303 placement_policy_util.cc:407] tablet d6c164a264eb4630912a74d65efb7387: detected majority of replicas (2 of 3) at location L0
I20240502 14:05:54.294397 14303 placement_policy_util.cc:407] tablet d6c164a264eb4630912a74d65efb7387: detected majority of replicas (2 of 3) at location L0
I20240502 14:05:55.265167 14308 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:55.266491 14308 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:55.266849 14308 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:55.301647 14303 placement_policy_util.cc:407] tablet d6c164a264eb4630912a74d65efb7387: detected majority of replicas (2 of 3) at location L0
I20240502 14:05:56.309036 14303 placement_policy_util.cc:407] tablet d6c164a264eb4630912a74d65efb7387: detected majority of replicas (2 of 3) at location L0
I20240502 14:05:57.267684 14308 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:57.268872 14308 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:57.269106 14308 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:57.316316 14303 placement_policy_util.cc:407] tablet d6c164a264eb4630912a74d65efb7387: detected majority of replicas (2 of 3) at location L0
I20240502 14:05:58.322631 14303 placement_policy_util.cc:407] tablet d6c164a264eb4630912a74d65efb7387: detected majority of replicas (2 of 3) at location L0
I20240502 14:05:59.269935 14308 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:05:59.271133 14308 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:05:59.271365 14308 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:05:59.328558 14303 placement_policy_util.cc:407] tablet d6c164a264eb4630912a74d65efb7387: detected majority of replicas (2 of 3) at location L0
I20240502 14:05:59.920414 11301 tablet_server.cc:178] TabletServer@127.11.9.65:0 shutting down...
I20240502 14:05:59.944571 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:05:59.945261 11301 tablet_replica.cc:330] T d6c164a264eb4630912a74d65efb7387 P cfd772dccab34b3ab0801ab5b4ccebc3: stopping tablet replica
I20240502 14:05:59.945785 11301 raft_consensus.cc:2229] T d6c164a264eb4630912a74d65efb7387 P cfd772dccab34b3ab0801ab5b4ccebc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:05:59.946252 11301 raft_consensus.cc:2258] T d6c164a264eb4630912a74d65efb7387 P cfd772dccab34b3ab0801ab5b4ccebc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:05:59.966181 11301 tablet_server.cc:194] TabletServer@127.11.9.65:0 shutdown complete.
I20240502 14:05:59.978884 11301 tablet_server.cc:178] TabletServer@127.11.9.66:0 shutting down...
I20240502 14:06:00.003381 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:06:00.004056 11301 tablet_replica.cc:330] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55: stopping tablet replica
I20240502 14:06:00.004566 11301 raft_consensus.cc:2229] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:00.005383 11301 raft_consensus.cc:2258] T d6c164a264eb4630912a74d65efb7387 P fc188f6a27d448a4847a7601c7c3ac55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:00.023939 11301 tablet_server.cc:194] TabletServer@127.11.9.66:0 shutdown complete.
I20240502 14:06:00.033447 11301 tablet_server.cc:178] TabletServer@127.11.9.67:0 shutting down...
I20240502 14:06:00.054347 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:06:00.054893 11301 tablet_replica.cc:330] T d6c164a264eb4630912a74d65efb7387 P 5dc04d5dffb14379bdec724eedf369c3: stopping tablet replica
I20240502 14:06:00.055437 11301 raft_consensus.cc:2229] T d6c164a264eb4630912a74d65efb7387 P 5dc04d5dffb14379bdec724eedf369c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:00.055889 11301 raft_consensus.cc:2258] T d6c164a264eb4630912a74d65efb7387 P 5dc04d5dffb14379bdec724eedf369c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:00.075107 11301 tablet_server.cc:194] TabletServer@127.11.9.67:0 shutdown complete.
I20240502 14:06:00.084416 11301 master.cc:537] Master@127.11.9.126:40313 shutting down...
I20240502 14:06:00.101267 11301 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ea9f0c56eb144f4fa405d3a86b75e4cd [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:00.101933 11301 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ea9f0c56eb144f4fa405d3a86b75e4cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:00.102247 11301 tablet_replica.cc:330] T 00000000000000000000000000000000 P ea9f0c56eb144f4fa405d3a86b75e4cd: stopping tablet replica
I20240502 14:06:00.119403 11301 master.cc:552] Master@127.11.9.126:40313 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (13015 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer-test.cc:516: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] AutoRebalancerTest.TestMaxMovesPerServer (6 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingUnstableCluster
I20240502 14:06:00.156769 11301 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.9.126:42185
I20240502 14:06:00.157830 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:00.163448 14585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:00.164069 14586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:00.164682 14588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:00.165297 11301 server_base.cc:1034] running on GCE node
I20240502 14:06:00.166115 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:06:00.166288 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:00.166414 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658760166402 us; error 0 us; skew 500 ppm
I20240502 14:06:00.166797 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:00.169068 11301 webserver.cc:460] Webserver started at http://127.11.9.126:33763/ using document root <none> and password file <none>
I20240502 14:06:00.169488 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:00.169668 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:00.169895 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:00.170933 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/master-0-root/instance:
uuid: "1107f0c04dfb45b49a30adaae5c56d68"
format_stamp: "Formatted at 2024-05-02 14:06:00 on dist-test-slave-j72s"
I20240502 14:06:00.175062 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240502 14:06:00.177982 14593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:00.178648 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240502 14:06:00.178860 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/master-0-root
uuid: "1107f0c04dfb45b49a30adaae5c56d68"
format_stamp: "Formatted at 2024-05-02 14:06:00 on dist-test-slave-j72s"
I20240502 14:06:00.179111 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:00.194540 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:00.195561 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:00.230330 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.126:42185
I20240502 14:06:00.230419 14644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.126:42185 every 8 connection(s)
I20240502 14:06:00.234050 14645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:00.244242 14645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1107f0c04dfb45b49a30adaae5c56d68: Bootstrap starting.
I20240502 14:06:00.248219 14645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1107f0c04dfb45b49a30adaae5c56d68: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:00.252071 14645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1107f0c04dfb45b49a30adaae5c56d68: No bootstrap required, opened a new log
I20240502 14:06:00.254053 14645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1107f0c04dfb45b49a30adaae5c56d68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1107f0c04dfb45b49a30adaae5c56d68" member_type: VOTER }
I20240502 14:06:00.254494 14645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1107f0c04dfb45b49a30adaae5c56d68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:00.254663 14645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1107f0c04dfb45b49a30adaae5c56d68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1107f0c04dfb45b49a30adaae5c56d68, State: Initialized, Role: FOLLOWER
I20240502 14:06:00.255162 14645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1107f0c04dfb45b49a30adaae5c56d68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1107f0c04dfb45b49a30adaae5c56d68" member_type: VOTER }
I20240502 14:06:00.255578 14645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1107f0c04dfb45b49a30adaae5c56d68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:06:00.255759 14645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1107f0c04dfb45b49a30adaae5c56d68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:06:00.255968 14645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1107f0c04dfb45b49a30adaae5c56d68 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:00.260010 14645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1107f0c04dfb45b49a30adaae5c56d68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1107f0c04dfb45b49a30adaae5c56d68" member_type: VOTER }
I20240502 14:06:00.260455 14645 leader_election.cc:302] T 00000000000000000000000000000000 P 1107f0c04dfb45b49a30adaae5c56d68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1107f0c04dfb45b49a30adaae5c56d68; no voters:
I20240502 14:06:00.261446 14645 leader_election.cc:288] T 00000000000000000000000000000000 P 1107f0c04dfb45b49a30adaae5c56d68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:06:00.261766 14648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1107f0c04dfb45b49a30adaae5c56d68 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:00.263041 14648 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1107f0c04dfb45b49a30adaae5c56d68 [term 1 LEADER]: Becoming Leader. State: Replica: 1107f0c04dfb45b49a30adaae5c56d68, State: Running, Role: LEADER
I20240502 14:06:00.263715 14648 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1107f0c04dfb45b49a30adaae5c56d68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1107f0c04dfb45b49a30adaae5c56d68" member_type: VOTER }
I20240502 14:06:00.264333 14645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1107f0c04dfb45b49a30adaae5c56d68 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:06:00.266808 14650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1107f0c04dfb45b49a30adaae5c56d68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1107f0c04dfb45b49a30adaae5c56d68. Latest consensus state: current_term: 1 leader_uuid: "1107f0c04dfb45b49a30adaae5c56d68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1107f0c04dfb45b49a30adaae5c56d68" member_type: VOTER } }
I20240502 14:06:00.266758 14649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1107f0c04dfb45b49a30adaae5c56d68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1107f0c04dfb45b49a30adaae5c56d68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1107f0c04dfb45b49a30adaae5c56d68" member_type: VOTER } }
I20240502 14:06:00.267480 14650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1107f0c04dfb45b49a30adaae5c56d68 [sys.catalog]: This master's current role is: LEADER
I20240502 14:06:00.267635 14649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1107f0c04dfb45b49a30adaae5c56d68 [sys.catalog]: This master's current role is: LEADER
I20240502 14:06:00.270807 14654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:06:00.278272 14654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:06:00.279186 11301 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:06:00.286641 14654 catalog_manager.cc:1347] Generated new cluster ID: 346e31ffc09945419057946799e6e3f5
I20240502 14:06:00.286899 14654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:06:00.324023 14654 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:06:00.325250 14654 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:06:00.341406 14654 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1107f0c04dfb45b49a30adaae5c56d68: Generated new TSK 0
I20240502 14:06:00.341995 14654 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:06:00.345727 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:00.351640 14666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:00.352785 14667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:00.353989 14669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:00.354759 11301 server_base.cc:1034] running on GCE node
I20240502 14:06:00.355484 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:06:00.355683 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:00.355839 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658760355819 us; error 0 us; skew 500 ppm
I20240502 14:06:00.356329 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:00.358606 11301 webserver.cc:460] Webserver started at http://127.11.9.65:37571/ using document root <none> and password file <none>
I20240502 14:06:00.359041 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:00.359220 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:00.359464 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:00.360535 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/ts-0-root/instance:
uuid: "52f1cdb77cab4395963618e9f92d67a2"
format_stamp: "Formatted at 2024-05-02 14:06:00 on dist-test-slave-j72s"
I20240502 14:06:00.364882 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 14:06:00.368124 14674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:00.368827 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:06:00.369079 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/ts-0-root
uuid: "52f1cdb77cab4395963618e9f92d67a2"
format_stamp: "Formatted at 2024-05-02 14:06:00 on dist-test-slave-j72s"
I20240502 14:06:00.369374 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:00.382253 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:00.383298 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:00.384673 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:00.386962 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:00.387153 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:00.387377 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:00.387540 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:00.440538 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.65:43237
I20240502 14:06:00.440562 14738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.65:43237 every 8 connection(s)
I20240502 14:06:00.447382 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:00.457839 14744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:00.459619 14743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:00.463521 11301 server_base.cc:1034] running on GCE node
I20240502 14:06:00.464594 14739 heartbeater.cc:347] Connected to a master server at 127.11.9.126:42185
W20240502 14:06:00.464870 14746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:00.464996 14739 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:00.465920 14739 heartbeater.cc:511] Master 127.11.9.126:42185 requested a full tablet report, sending...
I20240502 14:06:00.465941 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:06:00.466426 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:00.466581 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658760466565 us; error 0 us; skew 500 ppm
I20240502 14:06:00.467396 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:00.468770 14610 ts_manager.cc:194] Registered new tserver with Master: 52f1cdb77cab4395963618e9f92d67a2 (127.11.9.65:43237)
I20240502 14:06:00.471020 11301 webserver.cc:460] Webserver started at http://127.11.9.66:42683/ using document root <none> and password file <none>
I20240502 14:06:00.471836 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:00.472124 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:00.472241 14610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44444
I20240502 14:06:00.472589 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:00.474715 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/ts-1-root/instance:
uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc"
format_stamp: "Formatted at 2024-05-02 14:06:00 on dist-test-slave-j72s"
I20240502 14:06:00.479609 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:06:00.483165 14751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:00.483916 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:06:00.484182 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/ts-1-root
uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc"
format_stamp: "Formatted at 2024-05-02 14:06:00 on dist-test-slave-j72s"
I20240502 14:06:00.484429 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:00.511816 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:00.513087 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:00.514536 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:00.516907 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:00.517128 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:00.517365 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:00.517545 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20240502 14:06:00.556718 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.66:46509
I20240502 14:06:00.556816 14818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.66:46509 every 8 connection(s)
I20240502 14:06:00.561935 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:00.570279 14822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:00.570891 14823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:00.572484 14825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:00.573874 11301 server_base.cc:1034] running on GCE node
I20240502 14:06:00.574681 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:06:00.574868 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:00.575038 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658760575019 us; error 0 us; skew 500 ppm
I20240502 14:06:00.575564 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:00.577558 14819 heartbeater.cc:347] Connected to a master server at 127.11.9.126:42185
I20240502 14:06:00.577893 14819 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:00.578502 11301 webserver.cc:460] Webserver started at http://127.11.9.67:32999/ using document root <none> and password file <none>
I20240502 14:06:00.578619 14819 heartbeater.cc:511] Master 127.11.9.126:42185 requested a full tablet report, sending...
I20240502 14:06:00.579012 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:00.579228 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:00.579514 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:00.580689 14610 ts_manager.cc:194] Registered new tserver with Master: 9e9ae5d0e65c46c6a898ad76b6cc89bc (127.11.9.66:46509)
I20240502 14:06:00.580822 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/ts-2-root/instance:
uuid: "37dde43ea7234ce18548c4a574ec67cb"
format_stamp: "Formatted at 2024-05-02 14:06:00 on dist-test-slave-j72s"
I20240502 14:06:00.582104 14610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44456
I20240502 14:06:00.585377 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 14:06:00.588773 14830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:00.589457 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:06:00.589772 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/ts-2-root
uuid: "37dde43ea7234ce18548c4a574ec67cb"
format_stamp: "Formatted at 2024-05-02 14:06:00 on dist-test-slave-j72s"
I20240502 14:06:00.590027 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:00.622362 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:00.623528 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:00.624966 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:00.627175 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:00.627367 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:00.627593 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:00.627743 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:00.665278 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.67:38185
I20240502 14:06:00.665464 14892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.67:38185 every 8 connection(s)
I20240502 14:06:00.678208 14893 heartbeater.cc:347] Connected to a master server at 127.11.9.126:42185
I20240502 14:06:00.678586 14893 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:00.679265 14893 heartbeater.cc:511] Master 127.11.9.126:42185 requested a full tablet report, sending...
I20240502 14:06:00.681037 14610 ts_manager.cc:194] Registered new tserver with Master: 37dde43ea7234ce18548c4a574ec67cb (127.11.9.67:38185)
I20240502 14:06:00.681982 11301 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013592255s
I20240502 14:06:00.682538 14610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44464
I20240502 14:06:01.475479 14739 heartbeater.cc:503] Master 127.11.9.126:42185 was elected leader, sending a full tablet report...
I20240502 14:06:01.584404 14819 heartbeater.cc:503] Master 127.11.9.126:42185 was elected leader, sending a full tablet report...
I20240502 14:06:01.685356 14893 heartbeater.cc:503] Master 127.11.9.126:42185 was elected leader, sending a full tablet report...
I20240502 14:06:01.714291 11301 test_util.cc:268] Using random seed: 442968669
I20240502 14:06:01.736124 14610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44470:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:06:01.738516 14610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:06:01.794407 14701 tablet_service.cc:1467] Processing CreateTablet for tablet 05373a49f2a34ab9a96426e17bb2508c (DEFAULT_TABLE table=test-workload [id=e98a9bbbec574b5cb91c496490b12ccc]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240502 14:06:01.794783 14702 tablet_service.cc:1467] Processing CreateTablet for tablet a96b8cef09b04ac987749a0c749499f3 (DEFAULT_TABLE table=test-workload [id=e98a9bbbec574b5cb91c496490b12ccc]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240502 14:06:01.795982 14701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05373a49f2a34ab9a96426e17bb2508c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:01.795742 14700 tablet_service.cc:1467] Processing CreateTablet for tablet 21c5b3a203544940bc212be36ed25b1d (DEFAULT_TABLE table=test-workload [id=e98a9bbbec574b5cb91c496490b12ccc]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240502 14:06:01.797008 14700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21c5b3a203544940bc212be36ed25b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:01.798161 14857 tablet_service.cc:1467] Processing CreateTablet for tablet 05373a49f2a34ab9a96426e17bb2508c (DEFAULT_TABLE table=test-workload [id=e98a9bbbec574b5cb91c496490b12ccc]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240502 14:06:01.799418 14857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05373a49f2a34ab9a96426e17bb2508c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:01.800417 14856 tablet_service.cc:1467] Processing CreateTablet for tablet 21c5b3a203544940bc212be36ed25b1d (DEFAULT_TABLE table=test-workload [id=e98a9bbbec574b5cb91c496490b12ccc]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240502 14:06:01.801672 14856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21c5b3a203544940bc212be36ed25b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:01.802724 14783 tablet_service.cc:1467] Processing CreateTablet for tablet 05373a49f2a34ab9a96426e17bb2508c (DEFAULT_TABLE table=test-workload [id=e98a9bbbec574b5cb91c496490b12ccc]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240502 14:06:01.804064 14783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05373a49f2a34ab9a96426e17bb2508c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:01.797013 14858 tablet_service.cc:1467] Processing CreateTablet for tablet a96b8cef09b04ac987749a0c749499f3 (DEFAULT_TABLE table=test-workload [id=e98a9bbbec574b5cb91c496490b12ccc]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240502 14:06:01.807507 14858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a96b8cef09b04ac987749a0c749499f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:01.810735 14782 tablet_service.cc:1467] Processing CreateTablet for tablet 21c5b3a203544940bc212be36ed25b1d (DEFAULT_TABLE table=test-workload [id=e98a9bbbec574b5cb91c496490b12ccc]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240502 14:06:01.811939 14782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21c5b3a203544940bc212be36ed25b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:01.813275 14702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a96b8cef09b04ac987749a0c749499f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:01.801261 14784 tablet_service.cc:1467] Processing CreateTablet for tablet a96b8cef09b04ac987749a0c749499f3 (DEFAULT_TABLE table=test-workload [id=e98a9bbbec574b5cb91c496490b12ccc]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240502 14:06:01.816433 14784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a96b8cef09b04ac987749a0c749499f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:01.858211 14915 tablet_bootstrap.cc:492] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc: Bootstrap starting.
I20240502 14:06:01.863986 14914 tablet_bootstrap.cc:492] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb: Bootstrap starting.
I20240502 14:06:01.865774 14913 tablet_bootstrap.cc:492] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2: Bootstrap starting.
I20240502 14:06:01.866319 14915 tablet_bootstrap.cc:654] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:01.870780 14914 tablet_bootstrap.cc:654] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:01.870920 14915 tablet_bootstrap.cc:492] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc: No bootstrap required, opened a new log
I20240502 14:06:01.871457 14915 ts_tablet_manager.cc:1367] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20240502 14:06:01.872673 14913 tablet_bootstrap.cc:654] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:01.876564 14914 tablet_bootstrap.cc:492] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb: No bootstrap required, opened a new log
I20240502 14:06:01.875708 14915 raft_consensus.cc:348] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } }
I20240502 14:06:01.876878 14915 raft_consensus.cc:374] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:01.876993 14914 ts_tablet_manager.cc:1367] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20240502 14:06:01.877179 14915 raft_consensus.cc:729] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e9ae5d0e65c46c6a898ad76b6cc89bc, State: Initialized, Role: FOLLOWER
I20240502 14:06:01.877916 14915 consensus_queue.cc:260] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } }
I20240502 14:06:01.878260 14913 tablet_bootstrap.cc:492] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2: No bootstrap required, opened a new log
I20240502 14:06:01.878733 14913 ts_tablet_manager.cc:1367] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20240502 14:06:01.879678 14914 raft_consensus.cc:348] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } }
I20240502 14:06:01.880496 14914 raft_consensus.cc:374] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:01.880695 14915 ts_tablet_manager.cc:1398] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20240502 14:06:01.880802 14914 raft_consensus.cc:729] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37dde43ea7234ce18548c4a574ec67cb, State: Initialized, Role: FOLLOWER
I20240502 14:06:01.881755 14915 tablet_bootstrap.cc:492] T 05373a49f2a34ab9a96426e17bb2508c P 9e9ae5d0e65c46c6a898ad76b6cc89bc: Bootstrap starting.
I20240502 14:06:01.881675 14913 raft_consensus.cc:348] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } }
I20240502 14:06:01.881744 14914 consensus_queue.cc:260] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } }
I20240502 14:06:01.882555 14913 raft_consensus.cc:374] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:01.882874 14913 raft_consensus.cc:729] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f1cdb77cab4395963618e9f92d67a2, State: Initialized, Role: FOLLOWER
I20240502 14:06:01.883796 14913 consensus_queue.cc:260] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } }
I20240502 14:06:01.885046 14914 ts_tablet_manager.cc:1398] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb: Time spent starting tablet: real 0.008s user 0.006s sys 0.004s
I20240502 14:06:01.886238 14914 tablet_bootstrap.cc:492] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb: Bootstrap starting.
I20240502 14:06:01.892993 14913 ts_tablet_manager.cc:1398] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2: Time spent starting tablet: real 0.014s user 0.004s sys 0.009s
I20240502 14:06:01.893110 14914 tablet_bootstrap.cc:654] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:01.894061 14913 tablet_bootstrap.cc:492] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2: Bootstrap starting.
I20240502 14:06:01.896127 14919 raft_consensus.cc:482] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:01.896621 14919 raft_consensus.cc:504] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } }
I20240502 14:06:01.899263 14919 leader_election.cc:288] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37dde43ea7234ce18548c4a574ec67cb (127.11.9.67:38185), 52f1cdb77cab4395963618e9f92d67a2 (127.11.9.65:43237)
I20240502 14:06:01.900979 14913 tablet_bootstrap.cc:654] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:01.903597 14915 tablet_bootstrap.cc:654] T 05373a49f2a34ab9a96426e17bb2508c P 9e9ae5d0e65c46c6a898ad76b6cc89bc: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:01.904767 14914 tablet_bootstrap.cc:492] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb: No bootstrap required, opened a new log
I20240502 14:06:01.905351 14914 ts_tablet_manager.cc:1367] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.007s
I20240502 14:06:01.908548 14914 raft_consensus.cc:348] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } }
I20240502 14:06:01.909464 14914 raft_consensus.cc:374] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:01.909832 14914 raft_consensus.cc:729] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37dde43ea7234ce18548c4a574ec67cb, State: Initialized, Role: FOLLOWER
I20240502 14:06:01.910815 14914 consensus_queue.cc:260] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } }
I20240502 14:06:01.912942 14914 ts_tablet_manager.cc:1398] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 14:06:01.913170 14913 tablet_bootstrap.cc:492] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2: No bootstrap required, opened a new log
I20240502 14:06:01.913731 14913 ts_tablet_manager.cc:1367] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.007s
I20240502 14:06:01.916131 14914 tablet_bootstrap.cc:492] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb: Bootstrap starting.
I20240502 14:06:01.916251 14913 raft_consensus.cc:348] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } }
I20240502 14:06:01.917011 14913 raft_consensus.cc:374] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:01.917270 14913 raft_consensus.cc:729] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f1cdb77cab4395963618e9f92d67a2, State: Initialized, Role: FOLLOWER
I20240502 14:06:01.917987 14913 consensus_queue.cc:260] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } }
I20240502 14:06:01.921455 14913 ts_tablet_manager.cc:1398] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 14:06:01.922093 14914 tablet_bootstrap.cc:654] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:01.922315 14913 tablet_bootstrap.cc:492] T 05373a49f2a34ab9a96426e17bb2508c P 52f1cdb77cab4395963618e9f92d67a2: Bootstrap starting.
I20240502 14:06:01.928782 14913 tablet_bootstrap.cc:654] T 05373a49f2a34ab9a96426e17bb2508c P 52f1cdb77cab4395963618e9f92d67a2: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:01.934913 14914 tablet_bootstrap.cc:492] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb: No bootstrap required, opened a new log
I20240502 14:06:01.935384 14914 ts_tablet_manager.cc:1367] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.008s
I20240502 14:06:01.936761 14921 raft_consensus.cc:482] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:01.937203 14921 raft_consensus.cc:504] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } }
I20240502 14:06:01.937958 14914 raft_consensus.cc:348] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } }
I20240502 14:06:01.938669 14914 raft_consensus.cc:374] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:01.938943 14914 raft_consensus.cc:729] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37dde43ea7234ce18548c4a574ec67cb, State: Initialized, Role: FOLLOWER
I20240502 14:06:01.939728 14921 leader_election.cc:288] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e9ae5d0e65c46c6a898ad76b6cc89bc (127.11.9.66:46509), 37dde43ea7234ce18548c4a574ec67cb (127.11.9.67:38185)
I20240502 14:06:01.939581 14914 consensus_queue.cc:260] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } }
I20240502 14:06:01.940201 14921 raft_consensus.cc:482] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:01.940627 14921 raft_consensus.cc:504] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } }
I20240502 14:06:01.942492 14921 leader_election.cc:288] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37dde43ea7234ce18548c4a574ec67cb (127.11.9.67:38185), 9e9ae5d0e65c46c6a898ad76b6cc89bc (127.11.9.66:46509)
I20240502 14:06:01.943987 14914 ts_tablet_manager.cc:1398] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240502 14:06:01.944351 14913 tablet_bootstrap.cc:492] T 05373a49f2a34ab9a96426e17bb2508c P 52f1cdb77cab4395963618e9f92d67a2: No bootstrap required, opened a new log
I20240502 14:06:01.944831 14913 ts_tablet_manager.cc:1367] T 05373a49f2a34ab9a96426e17bb2508c P 52f1cdb77cab4395963618e9f92d67a2: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.004s
I20240502 14:06:01.947845 14913 raft_consensus.cc:348] T 05373a49f2a34ab9a96426e17bb2508c P 52f1cdb77cab4395963618e9f92d67a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } }
I20240502 14:06:01.948729 14913 raft_consensus.cc:374] T 05373a49f2a34ab9a96426e17bb2508c P 52f1cdb77cab4395963618e9f92d67a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:01.949049 14913 raft_consensus.cc:729] T 05373a49f2a34ab9a96426e17bb2508c P 52f1cdb77cab4395963618e9f92d67a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f1cdb77cab4395963618e9f92d67a2, State: Initialized, Role: FOLLOWER
I20240502 14:06:01.949815 14913 consensus_queue.cc:260] T 05373a49f2a34ab9a96426e17bb2508c P 52f1cdb77cab4395963618e9f92d67a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } }
I20240502 14:06:01.952700 14913 ts_tablet_manager.cc:1398] T 05373a49f2a34ab9a96426e17bb2508c P 52f1cdb77cab4395963618e9f92d67a2: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20240502 14:06:01.970503 14794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c5b3a203544940bc212be36ed25b1d" candidate_uuid: "52f1cdb77cab4395963618e9f92d67a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" is_pre_election: true
I20240502 14:06:01.972227 14793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96b8cef09b04ac987749a0c749499f3" candidate_uuid: "52f1cdb77cab4395963618e9f92d67a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" is_pre_election: true
I20240502 14:06:01.972977 14793 raft_consensus.cc:2454] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f1cdb77cab4395963618e9f92d67a2 in term 0.
W20240502 14:06:01.974238 14677 leader_election.cc:341] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9e9ae5d0e65c46c6a898ad76b6cc89bc (127.11.9.66:46509): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:01.975271 14677 leader_election.cc:302] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f1cdb77cab4395963618e9f92d67a2, 9e9ae5d0e65c46c6a898ad76b6cc89bc; no voters:
I20240502 14:06:01.976145 14929 raft_consensus.cc:2781] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:01.976528 14929 raft_consensus.cc:482] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:01.976840 14929 raft_consensus.cc:3037] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:01.986090 14712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96b8cef09b04ac987749a0c749499f3" candidate_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f1cdb77cab4395963618e9f92d67a2" is_pre_election: true
I20240502 14:06:01.986445 14915 tablet_bootstrap.cc:492] T 05373a49f2a34ab9a96426e17bb2508c P 9e9ae5d0e65c46c6a898ad76b6cc89bc: No bootstrap required, opened a new log
I20240502 14:06:01.987075 14915 ts_tablet_manager.cc:1367] T 05373a49f2a34ab9a96426e17bb2508c P 9e9ae5d0e65c46c6a898ad76b6cc89bc: Time spent bootstrapping tablet: real 0.106s user 0.011s sys 0.039s
I20240502 14:06:01.988518 14929 raft_consensus.cc:504] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } }
I20240502 14:06:01.989153 14915 raft_consensus.cc:348] T 05373a49f2a34ab9a96426e17bb2508c P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } }
I20240502 14:06:01.991956 14915 raft_consensus.cc:374] T 05373a49f2a34ab9a96426e17bb2508c P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:01.991906 14867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96b8cef09b04ac987749a0c749499f3" candidate_uuid: "52f1cdb77cab4395963618e9f92d67a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dde43ea7234ce18548c4a574ec67cb" is_pre_election: true
I20240502 14:06:01.992487 14712 raft_consensus.cc:2379] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e9ae5d0e65c46c6a898ad76b6cc89bc in current term 1: Already voted for candidate 52f1cdb77cab4395963618e9f92d67a2 in this term.
I20240502 14:06:01.992573 14915 raft_consensus.cc:729] T 05373a49f2a34ab9a96426e17bb2508c P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e9ae5d0e65c46c6a898ad76b6cc89bc, State: Initialized, Role: FOLLOWER
I20240502 14:06:01.992906 14867 raft_consensus.cc:2454] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f1cdb77cab4395963618e9f92d67a2 in term 0.
I20240502 14:06:01.990731 14920 raft_consensus.cc:482] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:01.993606 14915 consensus_queue.cc:260] T 05373a49f2a34ab9a96426e17bb2508c P 9e9ae5d0e65c46c6a898ad76b6cc89bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } }
I20240502 14:06:01.994139 14920 raft_consensus.cc:504] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } }
I20240502 14:06:01.996217 14915 ts_tablet_manager.cc:1398] T 05373a49f2a34ab9a96426e17bb2508c P 9e9ae5d0e65c46c6a898ad76b6cc89bc: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240502 14:06:01.997282 14915 tablet_bootstrap.cc:492] T 21c5b3a203544940bc212be36ed25b1d P 9e9ae5d0e65c46c6a898ad76b6cc89bc: Bootstrap starting.
I20240502 14:06:01.998286 14920 leader_election.cc:288] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f1cdb77cab4395963618e9f92d67a2 (127.11.9.65:43237), 9e9ae5d0e65c46c6a898ad76b6cc89bc (127.11.9.66:46509)
I20240502 14:06:01.991429 14868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c5b3a203544940bc212be36ed25b1d" candidate_uuid: "52f1cdb77cab4395963618e9f92d67a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dde43ea7234ce18548c4a574ec67cb" is_pre_election: true
I20240502 14:06:02.001262 14868 raft_consensus.cc:2454] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f1cdb77cab4395963618e9f92d67a2 in term 0.
I20240502 14:06:02.003193 14866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96b8cef09b04ac987749a0c749499f3" candidate_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dde43ea7234ce18548c4a574ec67cb" is_pre_election: true
I20240502 14:06:02.003871 14915 tablet_bootstrap.cc:654] T 21c5b3a203544940bc212be36ed25b1d P 9e9ae5d0e65c46c6a898ad76b6cc89bc: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:02.003886 14866 raft_consensus.cc:2454] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e9ae5d0e65c46c6a898ad76b6cc89bc in term 0.
I20240502 14:06:02.005637 14760 leader_election.cc:302] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dde43ea7234ce18548c4a574ec67cb, 9e9ae5d0e65c46c6a898ad76b6cc89bc; no voters:
I20240502 14:06:02.007870 14919 raft_consensus.cc:2781] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:02.008281 14919 raft_consensus.cc:482] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:02.008646 14919 raft_consensus.cc:3037] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:02.012786 14678 leader_election.cc:302] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37dde43ea7234ce18548c4a574ec67cb, 52f1cdb77cab4395963618e9f92d67a2; no voters: 9e9ae5d0e65c46c6a898ad76b6cc89bc
I20240502 14:06:02.014019 14929 leader_election.cc:288] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [CANDIDATE]: Term 1 election: Requested vote from peers 37dde43ea7234ce18548c4a574ec67cb (127.11.9.67:38185), 9e9ae5d0e65c46c6a898ad76b6cc89bc (127.11.9.66:46509)
I20240502 14:06:02.015938 14866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96b8cef09b04ac987749a0c749499f3" candidate_uuid: "52f1cdb77cab4395963618e9f92d67a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dde43ea7234ce18548c4a574ec67cb"
I20240502 14:06:02.017901 14866 raft_consensus.cc:3037] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:02.025338 14793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96b8cef09b04ac987749a0c749499f3" candidate_uuid: "52f1cdb77cab4395963618e9f92d67a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc"
I20240502 14:06:02.025697 14921 raft_consensus.cc:2781] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:02.026062 14921 raft_consensus.cc:482] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:02.026377 14921 raft_consensus.cc:3037] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:02.029151 14919 raft_consensus.cc:504] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } }
I20240502 14:06:02.030313 14793 raft_consensus.cc:2379] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 52f1cdb77cab4395963618e9f92d67a2 in current term 1: Already voted for candidate 9e9ae5d0e65c46c6a898ad76b6cc89bc in this term.
I20240502 14:06:02.032982 14921 raft_consensus.cc:504] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } }
I20240502 14:06:02.036280 14712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96b8cef09b04ac987749a0c749499f3" candidate_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f1cdb77cab4395963618e9f92d67a2"
I20240502 14:06:02.037099 14712 raft_consensus.cc:2379] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e9ae5d0e65c46c6a898ad76b6cc89bc in current term 1: Already voted for candidate 52f1cdb77cab4395963618e9f92d67a2 in this term.
I20240502 14:06:02.036967 14793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c5b3a203544940bc212be36ed25b1d" candidate_uuid: "52f1cdb77cab4395963618e9f92d67a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc"
I20240502 14:06:02.037726 14868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96b8cef09b04ac987749a0c749499f3" candidate_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dde43ea7234ce18548c4a574ec67cb"
W20240502 14:06:02.039338 14677 leader_election.cc:341] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9e9ae5d0e65c46c6a898ad76b6cc89bc (127.11.9.66:46509): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:06:02.040257 14760 leader_election.cc:302] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e9ae5d0e65c46c6a898ad76b6cc89bc; no voters: 37dde43ea7234ce18548c4a574ec67cb, 52f1cdb77cab4395963618e9f92d67a2
I20240502 14:06:02.041146 14921 leader_election.cc:288] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2 [CANDIDATE]: Term 1 election: Requested vote from peers 9e9ae5d0e65c46c6a898ad76b6cc89bc (127.11.9.66:46509), 37dde43ea7234ce18548c4a574ec67cb (127.11.9.67:38185)
I20240502 14:06:02.025097 14866 raft_consensus.cc:2454] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f1cdb77cab4395963618e9f92d67a2 in term 1.
I20240502 14:06:02.043794 14868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c5b3a203544940bc212be36ed25b1d" candidate_uuid: "52f1cdb77cab4395963618e9f92d67a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dde43ea7234ce18548c4a574ec67cb"
I20240502 14:06:02.044395 14868 raft_consensus.cc:3037] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:02.046540 14915 tablet_bootstrap.cc:492] T 21c5b3a203544940bc212be36ed25b1d P 9e9ae5d0e65c46c6a898ad76b6cc89bc: No bootstrap required, opened a new log
I20240502 14:06:02.047044 14915 ts_tablet_manager.cc:1367] T 21c5b3a203544940bc212be36ed25b1d P 9e9ae5d0e65c46c6a898ad76b6cc89bc: Time spent bootstrapping tablet: real 0.050s user 0.024s sys 0.024s
I20240502 14:06:02.049263 14915 raft_consensus.cc:348] T 21c5b3a203544940bc212be36ed25b1d P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } }
I20240502 14:06:02.049772 14915 raft_consensus.cc:374] T 21c5b3a203544940bc212be36ed25b1d P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:02.049991 14915 raft_consensus.cc:729] T 21c5b3a203544940bc212be36ed25b1d P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e9ae5d0e65c46c6a898ad76b6cc89bc, State: Initialized, Role: FOLLOWER
I20240502 14:06:02.049837 14793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05373a49f2a34ab9a96426e17bb2508c" candidate_uuid: "37dde43ea7234ce18548c4a574ec67cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" is_pre_election: true
I20240502 14:06:02.050593 14915 consensus_queue.cc:260] T 21c5b3a203544940bc212be36ed25b1d P 9e9ae5d0e65c46c6a898ad76b6cc89bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } }
I20240502 14:06:02.050565 14793 raft_consensus.cc:2454] T 05373a49f2a34ab9a96426e17bb2508c P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37dde43ea7234ce18548c4a574ec67cb in term 0.
I20240502 14:06:02.051903 14868 raft_consensus.cc:2454] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f1cdb77cab4395963618e9f92d67a2 in term 1.
I20240502 14:06:02.041262 14919 leader_election.cc:288] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [CANDIDATE]: Term 1 election: Requested vote from peers 37dde43ea7234ce18548c4a574ec67cb (127.11.9.67:38185), 52f1cdb77cab4395963618e9f92d67a2 (127.11.9.65:43237)
I20240502 14:06:02.041322 14935 raft_consensus.cc:2726] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:06:02.053040 14678 leader_election.cc:302] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37dde43ea7234ce18548c4a574ec67cb, 52f1cdb77cab4395963618e9f92d67a2; no voters: 9e9ae5d0e65c46c6a898ad76b6cc89bc
I20240502 14:06:02.052959 14833 leader_election.cc:302] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dde43ea7234ce18548c4a574ec67cb, 9e9ae5d0e65c46c6a898ad76b6cc89bc; no voters:
I20240502 14:06:02.054267 14920 raft_consensus.cc:2781] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:02.054371 14678 leader_election.cc:302] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37dde43ea7234ce18548c4a574ec67cb, 52f1cdb77cab4395963618e9f92d67a2; no voters: 9e9ae5d0e65c46c6a898ad76b6cc89bc
I20240502 14:06:02.054847 14920 raft_consensus.cc:482] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:02.055363 14920 raft_consensus.cc:3037] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:02.055536 14929 raft_consensus.cc:2781] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:02.056723 14921 raft_consensus.cc:2781] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:02.067706 14712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05373a49f2a34ab9a96426e17bb2508c" candidate_uuid: "37dde43ea7234ce18548c4a574ec67cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f1cdb77cab4395963618e9f92d67a2" is_pre_election: true
I20240502 14:06:02.068235 14712 raft_consensus.cc:2454] T 05373a49f2a34ab9a96426e17bb2508c P 52f1cdb77cab4395963618e9f92d67a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37dde43ea7234ce18548c4a574ec67cb in term 0.
I20240502 14:06:02.071381 14920 raft_consensus.cc:504] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } }
I20240502 14:06:02.074967 14712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05373a49f2a34ab9a96426e17bb2508c" candidate_uuid: "37dde43ea7234ce18548c4a574ec67cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f1cdb77cab4395963618e9f92d67a2"
I20240502 14:06:02.075207 14793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05373a49f2a34ab9a96426e17bb2508c" candidate_uuid: "37dde43ea7234ce18548c4a574ec67cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc"
I20240502 14:06:02.075695 14712 raft_consensus.cc:3037] T 05373a49f2a34ab9a96426e17bb2508c P 52f1cdb77cab4395963618e9f92d67a2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:02.076020 14793 raft_consensus.cc:3037] T 05373a49f2a34ab9a96426e17bb2508c P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:02.077242 14921 raft_consensus.cc:686] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2 [term 1 LEADER]: Becoming Leader. State: Replica: 52f1cdb77cab4395963618e9f92d67a2, State: Running, Role: LEADER
I20240502 14:06:02.078191 14921 consensus_queue.cc:237] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } }
I20240502 14:06:02.079402 14920 leader_election.cc:288] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [CANDIDATE]: Term 1 election: Requested vote from peers 52f1cdb77cab4395963618e9f92d67a2 (127.11.9.65:43237), 9e9ae5d0e65c46c6a898ad76b6cc89bc (127.11.9.66:46509)
I20240502 14:06:02.082281 14929 raft_consensus.cc:686] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [term 1 LEADER]: Becoming Leader. State: Replica: 52f1cdb77cab4395963618e9f92d67a2, State: Running, Role: LEADER
I20240502 14:06:02.083959 14929 consensus_queue.cc:237] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } }
I20240502 14:06:02.086642 14793 raft_consensus.cc:2454] T 05373a49f2a34ab9a96426e17bb2508c P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37dde43ea7234ce18548c4a574ec67cb in term 1.
I20240502 14:06:02.087733 14712 raft_consensus.cc:2454] T 05373a49f2a34ab9a96426e17bb2508c P 52f1cdb77cab4395963618e9f92d67a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37dde43ea7234ce18548c4a574ec67cb in term 1.
I20240502 14:06:02.087929 14833 leader_election.cc:302] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dde43ea7234ce18548c4a574ec67cb, 9e9ae5d0e65c46c6a898ad76b6cc89bc; no voters:
I20240502 14:06:02.089030 14915 ts_tablet_manager.cc:1398] T 21c5b3a203544940bc212be36ed25b1d P 9e9ae5d0e65c46c6a898ad76b6cc89bc: Time spent starting tablet: real 0.042s user 0.019s sys 0.022s
I20240502 14:06:02.089387 14920 raft_consensus.cc:2781] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:02.090871 14920 raft_consensus.cc:686] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [term 1 LEADER]: Becoming Leader. State: Replica: 37dde43ea7234ce18548c4a574ec67cb, State: Running, Role: LEADER
I20240502 14:06:02.091712 14920 consensus_queue.cc:237] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } }
I20240502 14:06:02.099133 14608 catalog_manager.cc:5484] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f1cdb77cab4395963618e9f92d67a2 (127.11.9.65). New cstate: current_term: 1 leader_uuid: "52f1cdb77cab4395963618e9f92d67a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:02.099303 14609 catalog_manager.cc:5484] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 37dde43ea7234ce18548c4a574ec67cb (127.11.9.67). New cstate: current_term: 1 leader_uuid: "37dde43ea7234ce18548c4a574ec67cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } health_report { overall_health: HEALTHY } } }
I20240502 14:06:02.110356 14608 catalog_manager.cc:5484] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f1cdb77cab4395963618e9f92d67a2 (127.11.9.65). New cstate: current_term: 1 leader_uuid: "52f1cdb77cab4395963618e9f92d67a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:02.163432 14937 raft_consensus.cc:482] T 21c5b3a203544940bc212be36ed25b1d P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:02.163954 14937 raft_consensus.cc:504] T 21c5b3a203544940bc212be36ed25b1d P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } }
I20240502 14:06:02.165560 14937 leader_election.cc:288] T 21c5b3a203544940bc212be36ed25b1d P 9e9ae5d0e65c46c6a898ad76b6cc89bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f1cdb77cab4395963618e9f92d67a2 (127.11.9.65:43237), 37dde43ea7234ce18548c4a574ec67cb (127.11.9.67:38185)
I20240502 14:06:02.166443 14712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c5b3a203544940bc212be36ed25b1d" candidate_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f1cdb77cab4395963618e9f92d67a2" is_pre_election: true
I20240502 14:06:02.166460 14868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c5b3a203544940bc212be36ed25b1d" candidate_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dde43ea7234ce18548c4a574ec67cb" is_pre_election: true
I20240502 14:06:02.167351 14868 raft_consensus.cc:2379] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e9ae5d0e65c46c6a898ad76b6cc89bc in current term 1: Already voted for candidate 52f1cdb77cab4395963618e9f92d67a2 in this term.
I20240502 14:06:02.168171 14760 leader_election.cc:302] T 21c5b3a203544940bc212be36ed25b1d P 9e9ae5d0e65c46c6a898ad76b6cc89bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e9ae5d0e65c46c6a898ad76b6cc89bc; no voters: 37dde43ea7234ce18548c4a574ec67cb, 52f1cdb77cab4395963618e9f92d67a2
I20240502 14:06:02.168725 14937 raft_consensus.cc:3037] T 21c5b3a203544940bc212be36ed25b1d P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:02.174005 14937 raft_consensus.cc:2726] T 21c5b3a203544940bc212be36ed25b1d P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:06:02.182304 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:02.188694 14944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:02.190284 14945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:02.190994 14947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:02.191802 11301 server_base.cc:1034] running on GCE node
I20240502 14:06:02.193054 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:06:02.193248 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:02.193426 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658762193410 us; error 0 us; skew 500 ppm
I20240502 14:06:02.193969 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:02.196204 11301 webserver.cc:460] Webserver started at http://127.11.9.68:35531/ using document root <none> and password file <none>
I20240502 14:06:02.196624 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:02.196789 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:02.197017 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:02.198205 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/ts-3-root/instance:
uuid: "44711711b74846eb857b89b137bcf058"
format_stamp: "Formatted at 2024-05-02 14:06:02 on dist-test-slave-j72s"
I20240502 14:06:02.202418 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:06:02.205494 14952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:02.206188 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:06:02.206491 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/ts-3-root
uuid: "44711711b74846eb857b89b137bcf058"
format_stamp: "Formatted at 2024-05-02 14:06:02 on dist-test-slave-j72s"
I20240502 14:06:02.206802 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:02.230633 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:02.231621 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:02.232859 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:02.234865 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:02.235051 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:02.235272 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:02.235427 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:02.271700 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.68:44799
I20240502 14:06:02.271780 15014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.68:44799 every 8 connection(s)
I20240502 14:06:02.279009 14663 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:06:02.288057 15015 heartbeater.cc:347] Connected to a master server at 127.11.9.126:42185
I20240502 14:06:02.288383 15015 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:02.289080 15015 heartbeater.cc:511] Master 127.11.9.126:42185 requested a full tablet report, sending...
I20240502 14:06:02.290836 14608 ts_manager.cc:194] Registered new tserver with Master: 44711711b74846eb857b89b137bcf058 (127.11.9.68:44799)
I20240502 14:06:02.292546 14608 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44478
I20240502 14:06:02.293939 14712 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a96b8cef09b04ac987749a0c749499f3"
dest_uuid: "52f1cdb77cab4395963618e9f92d67a2"
mode: GRACEFUL
new_leader_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc"
from {username='slave'} at 127.0.0.1:52734
I20240502 14:06:02.294631 14712 raft_consensus.cc:595] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [term 1 LEADER]: Received request to transfer leadership to TS 9e9ae5d0e65c46c6a898ad76b6cc89bc
I20240502 14:06:02.295653 14663 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240502 14:06:02.296011 14663 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:06:02.372546 14920 consensus_queue.cc:1035] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:06:02.378907 14942 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20240502 14:06:02.379236 14942 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20240502 14:06:02.406149 14929 consensus_queue.cc:1035] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:02.413928 14921 consensus_queue.cc:1035] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:02.418675 14921 consensus_queue.cc:1035] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:02.444059 14921 consensus_queue.cc:1035] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:02.458601 14920 consensus_queue.cc:1035] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:02.886157 14921 raft_consensus.cc:979] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2: : Instructing follower 9e9ae5d0e65c46c6a898ad76b6cc89bc to start an election
I20240502 14:06:02.886683 14929 raft_consensus.cc:1067] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [term 1 LEADER]: Signalling peer 9e9ae5d0e65c46c6a898ad76b6cc89bc to start an election
I20240502 14:06:02.888285 14793 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a96b8cef09b04ac987749a0c749499f3"
dest_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc"
from {username='slave'} at 127.0.0.1:32882
I20240502 14:06:02.888806 14793 raft_consensus.cc:482] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:06:02.889106 14793 raft_consensus.cc:3037] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:02.893466 14793 raft_consensus.cc:504] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } }
I20240502 14:06:02.894973 14793 leader_election.cc:288] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [CANDIDATE]: Term 2 election: Requested vote from peers 37dde43ea7234ce18548c4a574ec67cb (127.11.9.67:38185), 52f1cdb77cab4395963618e9f92d67a2 (127.11.9.65:43237)
I20240502 14:06:02.896076 14712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96b8cef09b04ac987749a0c749499f3" candidate_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "52f1cdb77cab4395963618e9f92d67a2"
I20240502 14:06:02.896076 14866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96b8cef09b04ac987749a0c749499f3" candidate_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "37dde43ea7234ce18548c4a574ec67cb"
I20240502 14:06:02.896754 14712 raft_consensus.cc:3032] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:06:02.896795 14866 raft_consensus.cc:3037] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:02.897244 14712 raft_consensus.cc:729] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 52f1cdb77cab4395963618e9f92d67a2, State: Running, Role: LEADER
I20240502 14:06:02.898131 14712 consensus_queue.cc:260] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } }
I20240502 14:06:02.899366 14712 raft_consensus.cc:3037] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:02.903293 14866 raft_consensus.cc:2454] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e9ae5d0e65c46c6a898ad76b6cc89bc in term 2.
I20240502 14:06:02.904320 14760 leader_election.cc:302] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dde43ea7234ce18548c4a574ec67cb, 9e9ae5d0e65c46c6a898ad76b6cc89bc; no voters:
I20240502 14:06:02.905086 14937 raft_consensus.cc:2781] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:06:02.905622 14712 raft_consensus.cc:2454] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e9ae5d0e65c46c6a898ad76b6cc89bc in term 2.
I20240502 14:06:02.906172 14937 raft_consensus.cc:686] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 2 LEADER]: Becoming Leader. State: Replica: 9e9ae5d0e65c46c6a898ad76b6cc89bc, State: Running, Role: LEADER
I20240502 14:06:02.906973 14937 consensus_queue.cc:237] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } }
I20240502 14:06:02.912277 14608 catalog_manager.cc:5484] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc reported cstate change: term changed from 1 to 2, leader changed from 52f1cdb77cab4395963618e9f92d67a2 (127.11.9.65) to 9e9ae5d0e65c46c6a898ad76b6cc89bc (127.11.9.66). New cstate: current_term: 2 leader_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } health_report { overall_health: HEALTHY } } }
I20240502 14:06:03.203712 14866 raft_consensus.cc:1261] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [term 2 FOLLOWER]: Refusing update from remote peer 9e9ae5d0e65c46c6a898ad76b6cc89bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:06:03.204844 14937 consensus_queue.cc:1035] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:03.225917 14712 raft_consensus.cc:1261] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [term 2 FOLLOWER]: Refusing update from remote peer 9e9ae5d0e65c46c6a898ad76b6cc89bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:06:03.226971 14937 consensus_queue.cc:1035] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:03.294924 15015 heartbeater.cc:503] Master 127.11.9.126:42185 was elected leader, sending a full tablet report...
I20240502 14:06:03.300202 14793 consensus_queue.cc:237] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } attrs { replace: true } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: true } }
I20240502 14:06:03.305742 14712 raft_consensus.cc:1261] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [term 2 FOLLOWER]: Refusing update from remote peer 9e9ae5d0e65c46c6a898ad76b6cc89bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:06:03.306530 14866 raft_consensus.cc:1261] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [term 2 FOLLOWER]: Refusing update from remote peer 9e9ae5d0e65c46c6a898ad76b6cc89bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:06:03.307231 14937 consensus_queue.cc:1035] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:03.308065 15038 consensus_queue.cc:1035] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 14:06:03.316810 14752 consensus_peers.cc:485] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc -> Peer 44711711b74846eb857b89b137bcf058 (127.11.9.68:44799): Couldn't send request to peer 44711711b74846eb857b89b137bcf058. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a96b8cef09b04ac987749a0c749499f3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:03.316233 14712 raft_consensus.cc:2932] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 44711711b74846eb857b89b137bcf058 (127.11.9.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } attrs { replace: true } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: true } } }
I20240502 14:06:03.315796 14866 raft_consensus.cc:2932] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 44711711b74846eb857b89b137bcf058 (127.11.9.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } attrs { replace: true } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: true } } }
I20240502 14:06:03.326413 14608 catalog_manager.cc:5484] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb reported cstate change: config changed from index -1 to 3, NON_VOTER 44711711b74846eb857b89b137bcf058 (127.11.9.68) added. New cstate: current_term: 2 leader_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } attrs { replace: true } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: true } } }
I20240502 14:06:03.628779 15043 ts_tablet_manager.cc:902] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058: Initiating tablet copy from peer 9e9ae5d0e65c46c6a898ad76b6cc89bc (127.11.9.66:46509)
I20240502 14:06:03.631764 15043 tablet_copy_client.cc:308] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058: tablet copy: Beginning tablet copy session from remote peer at address 127.11.9.66:46509
I20240502 14:06:03.640266 14804 tablet_copy_service.cc:140] P 9e9ae5d0e65c46c6a898ad76b6cc89bc: Received BeginTabletCopySession request for tablet a96b8cef09b04ac987749a0c749499f3 from peer 44711711b74846eb857b89b137bcf058 ({username='slave'} at 127.0.0.1:32896)
I20240502 14:06:03.640743 14804 tablet_copy_service.cc:161] P 9e9ae5d0e65c46c6a898ad76b6cc89bc: Beginning new tablet copy session on tablet a96b8cef09b04ac987749a0c749499f3 from peer 44711711b74846eb857b89b137bcf058 at {username='slave'} at 127.0.0.1:32896: session id = 44711711b74846eb857b89b137bcf058-a96b8cef09b04ac987749a0c749499f3
I20240502 14:06:03.645789 14804 tablet_copy_source_session.cc:205] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:06:03.648161 15043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a96b8cef09b04ac987749a0c749499f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:03.650151 15042 raft_consensus.cc:2932] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 44711711b74846eb857b89b137bcf058 (127.11.9.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } attrs { replace: true } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: true } } }
I20240502 14:06:03.659441 15043 tablet_copy_client.cc:783] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058: tablet copy: Starting download of 0 data blocks...
I20240502 14:06:03.660037 15043 tablet_copy_client.cc:647] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:03.663453 15043 tablet_copy_client.cc:523] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:03.668843 14866 consensus_queue.cc:237] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: true } }
I20240502 14:06:03.669955 15043 tablet_bootstrap.cc:492] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058: Bootstrap starting.
I20240502 14:06:03.675905 14793 raft_consensus.cc:1261] T 05373a49f2a34ab9a96426e17bb2508c P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 1 FOLLOWER]: Refusing update from remote peer 37dde43ea7234ce18548c4a574ec67cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:03.678632 15051 consensus_queue.cc:1035] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:03.680753 14712 raft_consensus.cc:1261] T 05373a49f2a34ab9a96426e17bb2508c P 52f1cdb77cab4395963618e9f92d67a2 [term 1 FOLLOWER]: Refusing update from remote peer 37dde43ea7234ce18548c4a574ec67cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:03.682875 15051 consensus_queue.cc:1035] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:06:03.686570 14831 consensus_peers.cc:485] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb -> Peer 44711711b74846eb857b89b137bcf058 (127.11.9.68:44799): Couldn't send request to peer 44711711b74846eb857b89b137bcf058. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05373a49f2a34ab9a96426e17bb2508c. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:03.693378 15043 tablet_bootstrap.cc:492] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:06:03.694083 15043 tablet_bootstrap.cc:492] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058: Bootstrap complete.
I20240502 14:06:03.694571 15043 ts_tablet_manager.cc:1367] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058: Time spent bootstrapping tablet: real 0.025s user 0.020s sys 0.000s
I20240502 14:06:03.693670 14793 raft_consensus.cc:2932] T 05373a49f2a34ab9a96426e17bb2508c P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 44711711b74846eb857b89b137bcf058 (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: true } } }
I20240502 14:06:03.695397 14712 raft_consensus.cc:2932] T 05373a49f2a34ab9a96426e17bb2508c P 52f1cdb77cab4395963618e9f92d67a2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 44711711b74846eb857b89b137bcf058 (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: true } } }
I20240502 14:06:03.696408 15043 raft_consensus.cc:348] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [term 2 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } }
I20240502 14:06:03.697320 15043 raft_consensus.cc:729] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 44711711b74846eb857b89b137bcf058, State: Initialized, Role: LEARNER
I20240502 14:06:03.697863 15043 consensus_queue.cc:260] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } attrs { replace: true } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: true } }
I20240502 14:06:03.700506 15043 ts_tablet_manager.cc:1398] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20240502 14:06:03.702332 14804 tablet_copy_service.cc:342] P 9e9ae5d0e65c46c6a898ad76b6cc89bc: Request end of tablet copy session 44711711b74846eb857b89b137bcf058-a96b8cef09b04ac987749a0c749499f3 received from {username='slave'} at 127.0.0.1:32896
I20240502 14:06:03.702700 14804 tablet_copy_service.cc:433] P 9e9ae5d0e65c46c6a898ad76b6cc89bc: ending tablet copy session 44711711b74846eb857b89b137bcf058-a96b8cef09b04ac987749a0c749499f3 on tablet a96b8cef09b04ac987749a0c749499f3 with peer 44711711b74846eb857b89b137bcf058
I20240502 14:06:03.707360 14608 catalog_manager.cc:5484] T 05373a49f2a34ab9a96426e17bb2508c P 9e9ae5d0e65c46c6a898ad76b6cc89bc reported cstate change: config changed from index -1 to 2, NON_VOTER 44711711b74846eb857b89b137bcf058 (127.11.9.68) added. New cstate: current_term: 1 leader_uuid: "37dde43ea7234ce18548c4a574ec67cb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: true } } }
I20240502 14:06:03.766943 14942 raft_consensus.cc:2932] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 44711711b74846eb857b89b137bcf058 (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: true } } }
I20240502 14:06:03.977140 15043 ts_tablet_manager.cc:902] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058: Initiating tablet copy from peer 37dde43ea7234ce18548c4a574ec67cb (127.11.9.67:38185)
I20240502 14:06:03.980996 15043 tablet_copy_client.cc:308] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058: tablet copy: Beginning tablet copy session from remote peer at address 127.11.9.67:38185
I20240502 14:06:03.989928 14878 tablet_copy_service.cc:140] P 37dde43ea7234ce18548c4a574ec67cb: Received BeginTabletCopySession request for tablet 05373a49f2a34ab9a96426e17bb2508c from peer 44711711b74846eb857b89b137bcf058 ({username='slave'} at 127.0.0.1:45168)
I20240502 14:06:03.990420 14878 tablet_copy_service.cc:161] P 37dde43ea7234ce18548c4a574ec67cb: Beginning new tablet copy session on tablet 05373a49f2a34ab9a96426e17bb2508c from peer 44711711b74846eb857b89b137bcf058 at {username='slave'} at 127.0.0.1:45168: session id = 44711711b74846eb857b89b137bcf058-05373a49f2a34ab9a96426e17bb2508c
I20240502 14:06:03.995632 14990 raft_consensus.cc:1203] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240502 14:06:03.996433 14990 raft_consensus.cc:2932] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [term 2 LEARNER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 44711711b74846eb857b89b137bcf058 (127.11.9.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } attrs { replace: true } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: true } } }
I20240502 14:06:03.998750 14878 tablet_copy_source_session.cc:205] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:06:04.001847 15043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05373a49f2a34ab9a96426e17bb2508c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:04.018790 15043 tablet_copy_client.cc:783] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058: tablet copy: Starting download of 0 data blocks...
I20240502 14:06:04.019222 15043 tablet_copy_client.cc:647] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:04.022656 15043 tablet_copy_client.cc:523] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:04.029383 15043 tablet_bootstrap.cc:492] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058: Bootstrap starting.
I20240502 14:06:04.045982 15043 tablet_bootstrap.cc:492] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:04.046840 15043 tablet_bootstrap.cc:492] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058: Bootstrap complete.
I20240502 14:06:04.047358 15043 ts_tablet_manager.cc:1367] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.004s
I20240502 14:06:04.049679 15043 raft_consensus.cc:348] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: true } }
I20240502 14:06:04.050323 15043 raft_consensus.cc:729] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 44711711b74846eb857b89b137bcf058, State: Initialized, Role: LEARNER
I20240502 14:06:04.050936 15043 consensus_queue.cc:260] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: true } }
I20240502 14:06:04.052893 15043 ts_tablet_manager.cc:1398] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:06:04.054409 14878 tablet_copy_service.cc:342] P 37dde43ea7234ce18548c4a574ec67cb: Request end of tablet copy session 44711711b74846eb857b89b137bcf058-05373a49f2a34ab9a96426e17bb2508c received from {username='slave'} at 127.0.0.1:45168
I20240502 14:06:04.054738 14878 tablet_copy_service.cc:433] P 37dde43ea7234ce18548c4a574ec67cb: ending tablet copy session 44711711b74846eb857b89b137bcf058-05373a49f2a34ab9a96426e17bb2508c on tablet 05373a49f2a34ab9a96426e17bb2508c with peer 44711711b74846eb857b89b137bcf058
I20240502 14:06:04.296710 14663 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:06:04.299782 14663 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:06:04.300168 14663 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:06:04.328743 14989 raft_consensus.cc:1203] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:06:04.547734 14937 raft_consensus.cc:1050] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc: attempting to promote NON_VOTER 44711711b74846eb857b89b137bcf058 to VOTER
I20240502 14:06:04.549331 14937 consensus_queue.cc:237] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } attrs { replace: true } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } }
I20240502 14:06:04.554268 14989 raft_consensus.cc:1261] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [term 2 LEARNER]: Refusing update from remote peer 9e9ae5d0e65c46c6a898ad76b6cc89bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:06:04.554983 14712 raft_consensus.cc:1261] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [term 2 FOLLOWER]: Refusing update from remote peer 9e9ae5d0e65c46c6a898ad76b6cc89bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:06:04.555981 14866 raft_consensus.cc:1261] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [term 2 FOLLOWER]: Refusing update from remote peer 9e9ae5d0e65c46c6a898ad76b6cc89bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:06:04.556134 15065 consensus_queue.cc:1035] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 14:06:04.557592 15042 consensus_queue.cc:1035] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:04.558342 14937 consensus_queue.cc:1035] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:04.577308 14990 raft_consensus.cc:2932] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 44711711b74846eb857b89b137bcf058 (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } attrs { replace: true } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } } }
I20240502 14:06:04.579026 14712 raft_consensus.cc:2932] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 44711711b74846eb857b89b137bcf058 (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } attrs { replace: true } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } } }
I20240502 14:06:04.590514 14609 catalog_manager.cc:5484] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 reported cstate change: config changed from index 3 to 4, 44711711b74846eb857b89b137bcf058 (127.11.9.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } attrs { replace: true } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } } }
I20240502 14:06:04.624871 14942 raft_consensus.cc:1050] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb: attempting to promote NON_VOTER 44711711b74846eb857b89b137bcf058 to VOTER
I20240502 14:06:04.626570 14942 consensus_queue.cc:237] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } }
I20240502 14:06:04.631282 14990 raft_consensus.cc:1261] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058 [term 1 LEARNER]: Refusing update from remote peer 37dde43ea7234ce18548c4a574ec67cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:04.631841 14794 raft_consensus.cc:1261] T 05373a49f2a34ab9a96426e17bb2508c P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 1 FOLLOWER]: Refusing update from remote peer 37dde43ea7234ce18548c4a574ec67cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:04.633361 15070 consensus_queue.cc:1035] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:04.633899 14712 raft_consensus.cc:1261] T 05373a49f2a34ab9a96426e17bb2508c P 52f1cdb77cab4395963618e9f92d67a2 [term 1 FOLLOWER]: Refusing update from remote peer 37dde43ea7234ce18548c4a574ec67cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:04.634265 14942 consensus_queue.cc:1035] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:04.635665 15057 consensus_queue.cc:1035] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 14:06:04.643240 14712 raft_consensus.cc:2932] T 05373a49f2a34ab9a96426e17bb2508c P 52f1cdb77cab4395963618e9f92d67a2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 44711711b74846eb857b89b137bcf058 (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } } }
I20240502 14:06:04.646478 14990 raft_consensus.cc:2932] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 44711711b74846eb857b89b137bcf058 (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } } }
I20240502 14:06:04.753736 15057 raft_consensus.cc:2932] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 44711711b74846eb857b89b137bcf058 (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } } }
I20240502 14:06:04.761683 14793 raft_consensus.cc:2932] T 05373a49f2a34ab9a96426e17bb2508c P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 44711711b74846eb857b89b137bcf058 (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } attrs { replace: true } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } } }
I20240502 14:06:04.763439 14608 catalog_manager.cc:5484] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb reported cstate change: config changed from index 2 to 3, 44711711b74846eb857b89b137bcf058 (127.11.9.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "37dde43ea7234ce18548c4a574ec67cb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:06:04.824204 15065 raft_consensus.cc:2932] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 44711711b74846eb857b89b137bcf058 (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } attrs { replace: true } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } } }
I20240502 14:06:04.832736 14868 raft_consensus.cc:2932] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 44711711b74846eb857b89b137bcf058 (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } attrs { replace: true } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } } }
I20240502 14:06:04.896374 14868 consensus_queue.cc:237] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } }
I20240502 14:06:04.902787 14990 raft_consensus.cc:1261] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058 [term 1 FOLLOWER]: Refusing update from remote peer 37dde43ea7234ce18548c4a574ec67cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:04.903532 14794 consensus_queue.cc:237] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } }
I20240502 14:06:04.904028 14712 raft_consensus.cc:1261] T 05373a49f2a34ab9a96426e17bb2508c P 52f1cdb77cab4395963618e9f92d67a2 [term 1 FOLLOWER]: Refusing update from remote peer 37dde43ea7234ce18548c4a574ec67cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:04.904342 15070 consensus_queue.cc:1035] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:04.905664 14942 consensus_queue.cc:1035] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:04.909042 14868 raft_consensus.cc:1261] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [term 2 FOLLOWER]: Refusing update from remote peer 9e9ae5d0e65c46c6a898ad76b6cc89bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:06:04.909677 14989 raft_consensus.cc:1261] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [term 2 FOLLOWER]: Refusing update from remote peer 9e9ae5d0e65c46c6a898ad76b6cc89bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:06:04.910549 14937 consensus_queue.cc:1035] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:06:04.912566 15072 consensus_queue.cc:1035] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 14:06:04.913491 14989 raft_consensus.cc:2932] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9e9ae5d0e65c46c6a898ad76b6cc89bc (127.11.9.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } } }
I20240502 14:06:04.919454 14866 raft_consensus.cc:2932] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 52f1cdb77cab4395963618e9f92d67a2 (127.11.9.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } } }
I20240502 14:06:04.924808 14712 raft_consensus.cc:2932] T 05373a49f2a34ab9a96426e17bb2508c P 52f1cdb77cab4395963618e9f92d67a2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9e9ae5d0e65c46c6a898ad76b6cc89bc (127.11.9.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } } }
I20240502 14:06:04.921473 14990 raft_consensus.cc:2932] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 52f1cdb77cab4395963618e9f92d67a2 (127.11.9.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } } }
I20240502 14:06:04.925669 14610 catalog_manager.cc:5484] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058 reported cstate change: config changed from index 3 to 4, VOTER 9e9ae5d0e65c46c6a898ad76b6cc89bc (127.11.9.66) evicted. New cstate: current_term: 1 leader_uuid: "37dde43ea7234ce18548c4a574ec67cb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } } }
I20240502 14:06:04.938200 14609 catalog_manager.cc:5484] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb reported cstate change: config changed from index 4 to 5, VOTER 52f1cdb77cab4395963618e9f92d67a2 (127.11.9.65) evicted. New cstate: current_term: 2 leader_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } } }
I20240502 14:06:04.979763 14784 tablet_service.cc:1514] Processing DeleteTablet for tablet 05373a49f2a34ab9a96426e17bb2508c with delete_type TABLET_DATA_TOMBSTONED (TS 9e9ae5d0e65c46c6a898ad76b6cc89bc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32878
I20240502 14:06:04.984102 15074 tablet_replica.cc:330] T 05373a49f2a34ab9a96426e17bb2508c P 9e9ae5d0e65c46c6a898ad76b6cc89bc: stopping tablet replica
I20240502 14:06:04.984845 15074 raft_consensus.cc:2229] T 05373a49f2a34ab9a96426e17bb2508c P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:04.985489 15074 raft_consensus.cc:2258] T 05373a49f2a34ab9a96426e17bb2508c P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:04.988379 15074 ts_tablet_manager.cc:1875] T 05373a49f2a34ab9a96426e17bb2508c P 9e9ae5d0e65c46c6a898ad76b6cc89bc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:06:04.999740 15074 ts_tablet_manager.cc:1888] T 05373a49f2a34ab9a96426e17bb2508c P 9e9ae5d0e65c46c6a898ad76b6cc89bc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:06:05.000056 15074 log.cc:1196] T 05373a49f2a34ab9a96426e17bb2508c P 9e9ae5d0e65c46c6a898ad76b6cc89bc: Deleting WAL directory at /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/ts-1-root/wals/05373a49f2a34ab9a96426e17bb2508c
I20240502 14:06:05.001413 14596 catalog_manager.cc:4830] TS 9e9ae5d0e65c46c6a898ad76b6cc89bc (127.11.9.66:46509): tablet 05373a49f2a34ab9a96426e17bb2508c (table test-workload [id=e98a9bbbec574b5cb91c496490b12ccc]) successfully deleted
I20240502 14:06:05.094178 14700 tablet_service.cc:1514] Processing DeleteTablet for tablet a96b8cef09b04ac987749a0c749499f3 with delete_type TABLET_DATA_TOMBSTONED (TS 52f1cdb77cab4395963618e9f92d67a2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52708
I20240502 14:06:05.095889 15075 tablet_replica.cc:330] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2: stopping tablet replica
I20240502 14:06:05.096521 15075 raft_consensus.cc:2229] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:05.097167 15075 raft_consensus.cc:2258] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:05.100248 15075 ts_tablet_manager.cc:1875] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:06:05.110394 15075 ts_tablet_manager.cc:1888] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 14:06:05.110728 15075 log.cc:1196] T a96b8cef09b04ac987749a0c749499f3 P 52f1cdb77cab4395963618e9f92d67a2: Deleting WAL directory at /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714658659912502-11301-0/minicluster-data/ts-0-root/wals/a96b8cef09b04ac987749a0c749499f3
I20240502 14:06:05.112149 14595 catalog_manager.cc:4830] TS 52f1cdb77cab4395963618e9f92d67a2 (127.11.9.65:43237): tablet a96b8cef09b04ac987749a0c749499f3 (table test-workload [id=e98a9bbbec574b5cb91c496490b12ccc]) successfully deleted
I20240502 14:06:05.370409 15070 raft_consensus.cc:2932] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9e9ae5d0e65c46c6a898ad76b6cc89bc (127.11.9.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } } }
I20240502 14:06:05.376863 14597 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 05373a49f2a34ab9a96426e17bb2508c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:06:05.470558 15072 raft_consensus.cc:2932] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 52f1cdb77cab4395963618e9f92d67a2 (127.11.9.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } } }
I20240502 14:06:05.477890 14596 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a96b8cef09b04ac987749a0c749499f3 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:06:06.301008 14663 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:06:06.303481 14663 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:06:06.303807 14663 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:06:08.304643 14663 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:06:08.307082 14663 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:06:08.307435 14663 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:06:10.308686 14663 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:06:10.313899 14663 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:06:10.314590 14663 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:06:10.644510 15078 raft_consensus.cc:482] T 05373a49f2a34ab9a96426e17bb2508c P 52f1cdb77cab4395963618e9f92d67a2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37dde43ea7234ce18548c4a574ec67cb)
I20240502 14:06:10.645753 15078 raft_consensus.cc:504] T 05373a49f2a34ab9a96426e17bb2508c P 52f1cdb77cab4395963618e9f92d67a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } }
I20240502 14:06:10.650142 15078 leader_election.cc:288] T 05373a49f2a34ab9a96426e17bb2508c P 52f1cdb77cab4395963618e9f92d67a2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37dde43ea7234ce18548c4a574ec67cb (127.11.9.67:38185), 44711711b74846eb857b89b137bcf058 (127.11.9.68:44799)
I20240502 14:06:10.656919 15079 raft_consensus.cc:482] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e9ae5d0e65c46c6a898ad76b6cc89bc)
I20240502 14:06:10.657346 15079 raft_consensus.cc:504] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } }
I20240502 14:06:10.660298 15079 leader_election.cc:288] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 37dde43ea7234ce18548c4a574ec67cb (127.11.9.67:38185), 9e9ae5d0e65c46c6a898ad76b6cc89bc (127.11.9.66:46509)
I20240502 14:06:10.886176 15079 raft_consensus.cc:482] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37dde43ea7234ce18548c4a574ec67cb)
I20240502 14:06:10.887995 15079 raft_consensus.cc:504] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } }
I20240502 14:06:10.890831 15076 raft_consensus.cc:482] T 21c5b3a203544940bc212be36ed25b1d P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52f1cdb77cab4395963618e9f92d67a2)
I20240502 14:06:10.891645 15076 raft_consensus.cc:504] T 21c5b3a203544940bc212be36ed25b1d P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } }
I20240502 14:06:10.894418 15076 leader_election.cc:288] T 21c5b3a203544940bc212be36ed25b1d P 9e9ae5d0e65c46c6a898ad76b6cc89bc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52f1cdb77cab4395963618e9f92d67a2 (127.11.9.65:43237), 37dde43ea7234ce18548c4a574ec67cb (127.11.9.67:38185)
I20240502 14:06:10.908262 15084 raft_consensus.cc:482] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52f1cdb77cab4395963618e9f92d67a2)
I20240502 14:06:10.910845 14711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c5b3a203544940bc212be36ed25b1d" candidate_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52f1cdb77cab4395963618e9f92d67a2" is_pre_election: true
I20240502 14:06:10.913520 15079 leader_election.cc:288] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52f1cdb77cab4395963618e9f92d67a2 (127.11.9.65:43237), 37dde43ea7234ce18548c4a574ec67cb (127.11.9.67:38185)
I20240502 14:06:10.919627 14866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05373a49f2a34ab9a96426e17bb2508c" candidate_uuid: "52f1cdb77cab4395963618e9f92d67a2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37dde43ea7234ce18548c4a574ec67cb" is_pre_election: true
I20240502 14:06:10.921069 14868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96b8cef09b04ac987749a0c749499f3" candidate_uuid: "44711711b74846eb857b89b137bcf058" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37dde43ea7234ce18548c4a574ec67cb" is_pre_election: true
I20240502 14:06:10.921942 14868 raft_consensus.cc:2454] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44711711b74846eb857b89b137bcf058 in term 2.
W20240502 14:06:10.922361 14760 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.798s user 0.024s sys 0.022s
W20240502 14:06:10.922868 14760 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.806s user 0.025s sys 0.022s
I20240502 14:06:10.909231 15084 raft_consensus.cc:504] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } }
I20240502 14:06:10.923671 14866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05373a49f2a34ab9a96426e17bb2508c" candidate_uuid: "44711711b74846eb857b89b137bcf058" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37dde43ea7234ce18548c4a574ec67cb" is_pre_election: true
I20240502 14:06:10.929075 15082 raft_consensus.cc:482] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e9ae5d0e65c46c6a898ad76b6cc89bc)
I20240502 14:06:10.929881 15082 raft_consensus.cc:504] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } }
I20240502 14:06:10.933039 15082 leader_election.cc:288] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e9ae5d0e65c46c6a898ad76b6cc89bc (127.11.9.66:46509), 44711711b74846eb857b89b137bcf058 (127.11.9.68:44799)
I20240502 14:06:10.936774 14956 leader_election.cc:302] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dde43ea7234ce18548c4a574ec67cb, 44711711b74846eb857b89b137bcf058; no voters:
I20240502 14:06:10.938077 15079 raft_consensus.cc:2781] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 14:06:10.938608 15079 raft_consensus.cc:482] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9e9ae5d0e65c46c6a898ad76b6cc89bc)
I20240502 14:06:10.939108 15079 raft_consensus.cc:3037] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:06:10.940838 14793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96b8cef09b04ac987749a0c749499f3" candidate_uuid: "44711711b74846eb857b89b137bcf058" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" is_pre_election: true
I20240502 14:06:10.942257 14794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96b8cef09b04ac987749a0c749499f3" candidate_uuid: "37dde43ea7234ce18548c4a574ec67cb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" is_pre_election: true
I20240502 14:06:10.947625 14990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96b8cef09b04ac987749a0c749499f3" candidate_uuid: "37dde43ea7234ce18548c4a574ec67cb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "44711711b74846eb857b89b137bcf058" is_pre_election: true
I20240502 14:06:10.952728 14866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c5b3a203544940bc212be36ed25b1d" candidate_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37dde43ea7234ce18548c4a574ec67cb" is_pre_election: true
I20240502 14:06:10.953140 14831 leader_election.cc:302] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37dde43ea7234ce18548c4a574ec67cb; no voters: 44711711b74846eb857b89b137bcf058, 9e9ae5d0e65c46c6a898ad76b6cc89bc
I20240502 14:06:10.964876 15081 raft_consensus.cc:2726] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:06:10.968478 14794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c5b3a203544940bc212be36ed25b1d" candidate_uuid: "37dde43ea7234ce18548c4a574ec67cb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" is_pre_election: true
I20240502 14:06:10.972249 14753 leader_election.cc:302] T 21c5b3a203544940bc212be36ed25b1d P 9e9ae5d0e65c46c6a898ad76b6cc89bc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e9ae5d0e65c46c6a898ad76b6cc89bc; no voters: 37dde43ea7234ce18548c4a574ec67cb, 52f1cdb77cab4395963618e9f92d67a2
I20240502 14:06:10.970857 15084 leader_election.cc:288] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52f1cdb77cab4395963618e9f92d67a2 (127.11.9.65:43237), 9e9ae5d0e65c46c6a898ad76b6cc89bc (127.11.9.66:46509)
I20240502 14:06:10.973263 15076 raft_consensus.cc:2726] T 21c5b3a203544940bc212be36ed25b1d P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:06:10.974082 14712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c5b3a203544940bc212be36ed25b1d" candidate_uuid: "37dde43ea7234ce18548c4a574ec67cb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52f1cdb77cab4395963618e9f92d67a2" is_pre_election: true
I20240502 14:06:10.976800 14832 leader_election.cc:302] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37dde43ea7234ce18548c4a574ec67cb; no voters: 52f1cdb77cab4395963618e9f92d67a2, 9e9ae5d0e65c46c6a898ad76b6cc89bc
I20240502 14:06:10.977936 15084 raft_consensus.cc:2726] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:06:10.978579 15079 raft_consensus.cc:504] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } }
I20240502 14:06:10.980672 15079 leader_election.cc:288] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [CANDIDATE]: Term 3 election: Requested vote from peers 37dde43ea7234ce18548c4a574ec67cb (127.11.9.67:38185), 9e9ae5d0e65c46c6a898ad76b6cc89bc (127.11.9.66:46509)
I20240502 14:06:10.981555 14867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96b8cef09b04ac987749a0c749499f3" candidate_uuid: "44711711b74846eb857b89b137bcf058" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37dde43ea7234ce18548c4a574ec67cb"
I20240502 14:06:10.982573 14794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96b8cef09b04ac987749a0c749499f3" candidate_uuid: "44711711b74846eb857b89b137bcf058" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc"
I20240502 14:06:10.984675 14956 leader_election.cc:302] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44711711b74846eb857b89b137bcf058; no voters: 37dde43ea7234ce18548c4a574ec67cb, 9e9ae5d0e65c46c6a898ad76b6cc89bc
I20240502 14:06:10.989558 14712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05373a49f2a34ab9a96426e17bb2508c" candidate_uuid: "44711711b74846eb857b89b137bcf058" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52f1cdb77cab4395963618e9f92d67a2" is_pre_election: true
I20240502 14:06:10.989797 14990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05373a49f2a34ab9a96426e17bb2508c" candidate_uuid: "52f1cdb77cab4395963618e9f92d67a2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "44711711b74846eb857b89b137bcf058" is_pre_election: true
I20240502 14:06:10.990768 14954 leader_election.cc:302] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44711711b74846eb857b89b137bcf058; no voters: 37dde43ea7234ce18548c4a574ec67cb, 52f1cdb77cab4395963618e9f92d67a2
I20240502 14:06:10.991289 15079 raft_consensus.cc:2726] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 14:06:10.991587 14675 leader_election.cc:302] T 05373a49f2a34ab9a96426e17bb2508c P 52f1cdb77cab4395963618e9f92d67a2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52f1cdb77cab4395963618e9f92d67a2; no voters: 37dde43ea7234ce18548c4a574ec67cb, 44711711b74846eb857b89b137bcf058
I20240502 14:06:10.992053 15079 raft_consensus.cc:2726] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:06:10.992987 15080 raft_consensus.cc:2726] T 05373a49f2a34ab9a96426e17bb2508c P 52f1cdb77cab4395963618e9f92d67a2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:06:11.088984 14815 debug-util.cc:398] Leaking SignalData structure 0x7b0800052200 after lost signal to thread 11304
W20240502 14:06:11.090183 14815 debug-util.cc:398] Leaking SignalData structure 0x7b08001b4e00 after lost signal to thread 14644
W20240502 14:06:11.092285 14815 debug-util.cc:398] Leaking SignalData structure 0x7b080013aa60 after lost signal to thread 14818
W20240502 14:06:11.093256 14815 debug-util.cc:398] Leaking SignalData structure 0x7b08000c1e80 after lost signal to thread 14892
I20240502 14:06:11.166210 14990 raft_consensus.cc:1226] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [term 3 FOLLOWER]: Rejecting Update request from peer 9e9ae5d0e65c46c6a898ad76b6cc89bc for earlier term 2. Current term is 3. Ops: []
I20240502 14:06:11.167939 15076 consensus_queue.cc:1046] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240502 14:06:11.228538 15089 raft_consensus.cc:3032] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:06:11.229034 15089 raft_consensus.cc:729] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9e9ae5d0e65c46c6a898ad76b6cc89bc, State: Running, Role: LEADER
I20240502 14:06:11.230209 15089 consensus_queue.cc:260] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } }
I20240502 14:06:11.231850 15089 raft_consensus.cc:3037] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:06:11.942099 15090 raft_consensus.cc:482] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:11.942521 15090 raft_consensus.cc:504] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } }
I20240502 14:06:11.944075 15090 leader_election.cc:288] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 37dde43ea7234ce18548c4a574ec67cb (127.11.9.67:38185), 9e9ae5d0e65c46c6a898ad76b6cc89bc (127.11.9.66:46509)
I20240502 14:06:11.944936 14867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96b8cef09b04ac987749a0c749499f3" candidate_uuid: "44711711b74846eb857b89b137bcf058" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37dde43ea7234ce18548c4a574ec67cb" is_pre_election: true
I20240502 14:06:11.945145 14794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96b8cef09b04ac987749a0c749499f3" candidate_uuid: "44711711b74846eb857b89b137bcf058" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" is_pre_election: true
I20240502 14:06:11.945448 14867 raft_consensus.cc:2454] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44711711b74846eb857b89b137bcf058 in term 2.
I20240502 14:06:11.945890 14794 raft_consensus.cc:2454] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44711711b74846eb857b89b137bcf058 in term 3.
I20240502 14:06:11.946431 14956 leader_election.cc:302] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dde43ea7234ce18548c4a574ec67cb, 44711711b74846eb857b89b137bcf058; no voters:
I20240502 14:06:11.947113 15090 raft_consensus.cc:2781] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 14:06:11.947440 15090 raft_consensus.cc:482] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:11.947678 15090 raft_consensus.cc:3037] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:06:11.952405 15090 raft_consensus.cc:504] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } }
I20240502 14:06:11.954021 15090 leader_election.cc:288] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [CANDIDATE]: Term 4 election: Requested vote from peers 37dde43ea7234ce18548c4a574ec67cb (127.11.9.67:38185), 9e9ae5d0e65c46c6a898ad76b6cc89bc (127.11.9.66:46509)
I20240502 14:06:11.954664 14867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96b8cef09b04ac987749a0c749499f3" candidate_uuid: "44711711b74846eb857b89b137bcf058" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "37dde43ea7234ce18548c4a574ec67cb"
I20240502 14:06:11.955027 14794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96b8cef09b04ac987749a0c749499f3" candidate_uuid: "44711711b74846eb857b89b137bcf058" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc"
I20240502 14:06:11.955233 14867 raft_consensus.cc:3037] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [term 2 FOLLOWER]: Advancing to term 4
I20240502 14:06:11.955610 14794 raft_consensus.cc:3037] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:06:11.959856 14867 raft_consensus.cc:2454] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44711711b74846eb857b89b137bcf058 in term 4.
I20240502 14:06:11.960723 14956 leader_election.cc:302] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dde43ea7234ce18548c4a574ec67cb, 44711711b74846eb857b89b137bcf058; no voters:
I20240502 14:06:11.961300 15090 raft_consensus.cc:2781] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:06:11.962306 15090 raft_consensus.cc:686] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [term 4 LEADER]: Becoming Leader. State: Replica: 44711711b74846eb857b89b137bcf058, State: Running, Role: LEADER
I20240502 14:06:11.962899 14794 raft_consensus.cc:2454] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44711711b74846eb857b89b137bcf058 in term 4.
I20240502 14:06:11.963357 15090 consensus_queue.cc:237] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } }
I20240502 14:06:11.975152 14607 catalog_manager.cc:5484] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 reported cstate change: term changed from 2 to 4, leader changed from 9e9ae5d0e65c46c6a898ad76b6cc89bc (127.11.9.66) to 44711711b74846eb857b89b137bcf058 (127.11.9.68). New cstate: current_term: 4 leader_uuid: "44711711b74846eb857b89b137bcf058" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44711711b74846eb857b89b137bcf058" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 44799 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:06:12.198940 14867 raft_consensus.cc:1261] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [term 4 FOLLOWER]: Refusing update from remote peer 44711711b74846eb857b89b137bcf058: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240502 14:06:12.200156 15090 consensus_queue.cc:1035] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:06:12.241413 14794 raft_consensus.cc:1261] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 4 FOLLOWER]: Refusing update from remote peer 44711711b74846eb857b89b137bcf058: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240502 14:06:12.242733 15090 consensus_queue.cc:1035] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:06:12.315946 14663 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:06:12.318742 14663 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:06:12.319177 14663 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:06:12.921598 11301 tablet_server.cc:178] TabletServer@127.11.9.65:0 shutting down...
I20240502 14:06:12.949225 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:06:12.950098 11301 tablet_replica.cc:330] T 05373a49f2a34ab9a96426e17bb2508c P 52f1cdb77cab4395963618e9f92d67a2: stopping tablet replica
I20240502 14:06:12.950719 11301 raft_consensus.cc:2229] T 05373a49f2a34ab9a96426e17bb2508c P 52f1cdb77cab4395963618e9f92d67a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:12.951668 11301 raft_consensus.cc:2258] T 05373a49f2a34ab9a96426e17bb2508c P 52f1cdb77cab4395963618e9f92d67a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:12.954084 11301 tablet_replica.cc:330] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2: stopping tablet replica
I20240502 14:06:12.954545 11301 raft_consensus.cc:2229] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:12.955412 11301 raft_consensus.cc:2258] T 21c5b3a203544940bc212be36ed25b1d P 52f1cdb77cab4395963618e9f92d67a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:12.979741 11301 tablet_server.cc:194] TabletServer@127.11.9.65:0 shutdown complete.
I20240502 14:06:12.995653 11301 tablet_server.cc:178] TabletServer@127.11.9.66:0 shutting down...
W20240502 14:06:13.004403 14832 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.9.65:43237: connect: Connection refused (error 111)
W20240502 14:06:13.010672 14832 consensus_peers.cc:485] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb -> Peer 52f1cdb77cab4395963618e9f92d67a2 (127.11.9.65:43237): Couldn't send request to peer 52f1cdb77cab4395963618e9f92d67a2. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.65:43237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:13.027995 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:06:13.028899 11301 tablet_replica.cc:330] T 21c5b3a203544940bc212be36ed25b1d P 9e9ae5d0e65c46c6a898ad76b6cc89bc: stopping tablet replica
I20240502 14:06:13.029819 11301 raft_consensus.cc:2229] T 21c5b3a203544940bc212be36ed25b1d P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:13.030352 11301 raft_consensus.cc:2258] T 21c5b3a203544940bc212be36ed25b1d P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:13.033464 11301 tablet_replica.cc:330] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc: stopping tablet replica
I20240502 14:06:13.034190 11301 raft_consensus.cc:2229] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:13.034698 11301 raft_consensus.cc:2258] T a96b8cef09b04ac987749a0c749499f3 P 9e9ae5d0e65c46c6a898ad76b6cc89bc [term 4 FOLLOWER]: Raft consensus is shut down!
W20240502 14:06:13.104058 14955 consensus_peers.cc:485] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 -> Peer 9e9ae5d0e65c46c6a898ad76b6cc89bc (127.11.9.66:46509): Couldn't send request to peer 9e9ae5d0e65c46c6a898ad76b6cc89bc. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.66:46509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:13.568970 15108 raft_consensus.cc:482] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52f1cdb77cab4395963618e9f92d67a2)
I20240502 14:06:13.569581 15108 raft_consensus.cc:504] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f1cdb77cab4395963618e9f92d67a2" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43237 } } peers { permanent_uuid: "9e9ae5d0e65c46c6a898ad76b6cc89bc" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 46509 } } peers { permanent_uuid: "37dde43ea7234ce18548c4a574ec67cb" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38185 } }
I20240502 14:06:13.571703 15108 leader_election.cc:288] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52f1cdb77cab4395963618e9f92d67a2 (127.11.9.65:43237), 9e9ae5d0e65c46c6a898ad76b6cc89bc (127.11.9.66:46509)
W20240502 14:06:13.576025 14832 leader_election.cc:334] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 52f1cdb77cab4395963618e9f92d67a2 (127.11.9.65:43237): Network error: Client connection negotiation failed: client connection to 127.11.9.65:43237: connect: Connection refused (error 111)
W20240502 14:06:13.577639 14833 leader_election.cc:334] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9e9ae5d0e65c46c6a898ad76b6cc89bc (127.11.9.66:46509): Network error: Client connection negotiation failed: client connection to 127.11.9.66:46509: connect: Connection refused (error 111)
I20240502 14:06:13.578125 14833 leader_election.cc:302] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37dde43ea7234ce18548c4a574ec67cb; no voters: 52f1cdb77cab4395963618e9f92d67a2, 9e9ae5d0e65c46c6a898ad76b6cc89bc
I20240502 14:06:13.578899 15108 raft_consensus.cc:2726] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:06:13.946461 11301 tablet_server.cc:194] TabletServer@127.11.9.66:0 shutdown complete.
I20240502 14:06:13.959580 11301 tablet_server.cc:178] TabletServer@127.11.9.67:0 shutting down...
I20240502 14:06:13.985914 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:06:13.986572 11301 tablet_replica.cc:330] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb: stopping tablet replica
I20240502 14:06:13.987169 11301 raft_consensus.cc:2229] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:13.987717 11301 raft_consensus.cc:2258] T a96b8cef09b04ac987749a0c749499f3 P 37dde43ea7234ce18548c4a574ec67cb [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:13.989720 11301 tablet_replica.cc:330] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb: stopping tablet replica
I20240502 14:06:13.990186 11301 raft_consensus.cc:2229] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:13.990610 11301 raft_consensus.cc:2258] T 21c5b3a203544940bc212be36ed25b1d P 37dde43ea7234ce18548c4a574ec67cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:13.992413 11301 tablet_replica.cc:330] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb: stopping tablet replica
I20240502 14:06:13.992935 11301 raft_consensus.cc:2229] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:13.993707 11301 raft_consensus.cc:2258] T 05373a49f2a34ab9a96426e17bb2508c P 37dde43ea7234ce18548c4a574ec67cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:14.015977 11301 tablet_server.cc:194] TabletServer@127.11.9.67:0 shutdown complete.
I20240502 14:06:14.031646 11301 tablet_server.cc:178] TabletServer@127.11.9.68:0 shutting down...
I20240502 14:06:14.053148 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:06:14.053803 11301 tablet_replica.cc:330] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058: stopping tablet replica
I20240502 14:06:14.054383 11301 raft_consensus.cc:2229] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:14.054879 11301 raft_consensus.cc:2258] T 05373a49f2a34ab9a96426e17bb2508c P 44711711b74846eb857b89b137bcf058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:14.056787 11301 tablet_replica.cc:330] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058: stopping tablet replica
I20240502 14:06:14.057236 11301 raft_consensus.cc:2229] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [term 4 LEADER]: Raft consensus shutting down.
I20240502 14:06:14.057978 11301 raft_consensus.cc:2258] T a96b8cef09b04ac987749a0c749499f3 P 44711711b74846eb857b89b137bcf058 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:14.077765 11301 tablet_server.cc:194] TabletServer@127.11.9.68:0 shutdown complete.
I20240502 14:06:14.088790 11301 master.cc:537] Master@127.11.9.126:42185 shutting down...
I20240502 14:06:14.105188 11301 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1107f0c04dfb45b49a30adaae5c56d68 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:14.105697 11301 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1107f0c04dfb45b49a30adaae5c56d68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:14.106014 11301 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1107f0c04dfb45b49a30adaae5c56d68: stopping tablet replica
I20240502 14:06:14.123463 11301 master.cc:552] Master@127.11.9.126:42185 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingUnstableCluster (14002 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor
I20240502 14:06:14.159719 11301 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.9.126:35053
I20240502 14:06:14.160713 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:14.166853 15112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:14.167351 15113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:14.168144 15115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:14.168679 11301 server_base.cc:1034] running on GCE node
I20240502 14:06:14.169505 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:06:14.169795 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:14.169909 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658774169897 us; error 0 us; skew 500 ppm
I20240502 14:06:14.170325 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:14.177824 11301 webserver.cc:460] Webserver started at http://127.11.9.126:38783/ using document root <none> and password file <none>
I20240502 14:06:14.178241 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:14.178400 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:14.178618 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:14.179685 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714658659912502-11301-0/minicluster-data/master-0-root/instance:
uuid: "8157ceecdb1348259d010953425dfc43"
format_stamp: "Formatted at 2024-05-02 14:06:14 on dist-test-slave-j72s"
I20240502 14:06:14.183863 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20240502 14:06:14.186980 15120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:14.187747 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:06:14.187992 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714658659912502-11301-0/minicluster-data/master-0-root
uuid: "8157ceecdb1348259d010953425dfc43"
format_stamp: "Formatted at 2024-05-02 14:06:14 on dist-test-slave-j72s"
I20240502 14:06:14.188243 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714658659912502-11301-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714658659912502-11301-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714658659912502-11301-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:14.203326 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:14.204434 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:14.240010 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.126:35053
I20240502 14:06:14.240110 15171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.126:35053 every 8 connection(s)
I20240502 14:06:14.243711 15172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:14.253273 15172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8157ceecdb1348259d010953425dfc43: Bootstrap starting.
I20240502 14:06:14.257201 15172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8157ceecdb1348259d010953425dfc43: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:14.261258 15172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8157ceecdb1348259d010953425dfc43: No bootstrap required, opened a new log
I20240502 14:06:14.263099 15172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8157ceecdb1348259d010953425dfc43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8157ceecdb1348259d010953425dfc43" member_type: VOTER }
I20240502 14:06:14.263464 15172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8157ceecdb1348259d010953425dfc43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:14.263689 15172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8157ceecdb1348259d010953425dfc43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8157ceecdb1348259d010953425dfc43, State: Initialized, Role: FOLLOWER
I20240502 14:06:14.264237 15172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8157ceecdb1348259d010953425dfc43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8157ceecdb1348259d010953425dfc43" member_type: VOTER }
I20240502 14:06:14.264716 15172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8157ceecdb1348259d010953425dfc43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:06:14.264918 15172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8157ceecdb1348259d010953425dfc43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:06:14.265151 15172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8157ceecdb1348259d010953425dfc43 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:14.269060 15172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8157ceecdb1348259d010953425dfc43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8157ceecdb1348259d010953425dfc43" member_type: VOTER }
I20240502 14:06:14.269546 15172 leader_election.cc:302] T 00000000000000000000000000000000 P 8157ceecdb1348259d010953425dfc43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8157ceecdb1348259d010953425dfc43; no voters:
I20240502 14:06:14.270561 15172 leader_election.cc:288] T 00000000000000000000000000000000 P 8157ceecdb1348259d010953425dfc43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:06:14.270944 15175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8157ceecdb1348259d010953425dfc43 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:14.272280 15175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8157ceecdb1348259d010953425dfc43 [term 1 LEADER]: Becoming Leader. State: Replica: 8157ceecdb1348259d010953425dfc43, State: Running, Role: LEADER
I20240502 14:06:14.272946 15175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8157ceecdb1348259d010953425dfc43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8157ceecdb1348259d010953425dfc43" member_type: VOTER }
I20240502 14:06:14.273306 15172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8157ceecdb1348259d010953425dfc43 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:06:14.277257 15176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8157ceecdb1348259d010953425dfc43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8157ceecdb1348259d010953425dfc43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8157ceecdb1348259d010953425dfc43" member_type: VOTER } }
I20240502 14:06:14.277341 15177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8157ceecdb1348259d010953425dfc43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8157ceecdb1348259d010953425dfc43. Latest consensus state: current_term: 1 leader_uuid: "8157ceecdb1348259d010953425dfc43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8157ceecdb1348259d010953425dfc43" member_type: VOTER } }
I20240502 14:06:14.278070 15176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8157ceecdb1348259d010953425dfc43 [sys.catalog]: This master's current role is: LEADER
I20240502 14:06:14.278143 15177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8157ceecdb1348259d010953425dfc43 [sys.catalog]: This master's current role is: LEADER
I20240502 14:06:14.280757 15182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:06:14.286213 15182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:06:14.288302 11301 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:06:14.294301 15182 catalog_manager.cc:1347] Generated new cluster ID: 901f2ccfc53249faa7d99d6179b4bc24
I20240502 14:06:14.294575 15182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:06:14.314363 15182 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:06:14.315816 15182 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:06:14.328784 15182 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8157ceecdb1348259d010953425dfc43: Generated new TSK 0
I20240502 14:06:14.329331 15182 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:06:14.355317 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:14.361310 15193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:14.362507 15194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:14.363960 11301 server_base.cc:1034] running on GCE node
W20240502 14:06:14.364722 15196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:14.365449 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:06:14.365689 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:14.365860 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658774365843 us; error 0 us; skew 500 ppm
I20240502 14:06:14.366365 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:14.368556 11301 webserver.cc:460] Webserver started at http://127.11.9.65:40589/ using document root <none> and password file <none>
I20240502 14:06:14.368981 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:14.369165 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:14.369405 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:14.370457 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714658659912502-11301-0/minicluster-data/ts-0-root/instance:
uuid: "1f4a05af65de4ff880d46dc181844eb7"
format_stamp: "Formatted at 2024-05-02 14:06:14 on dist-test-slave-j72s"
I20240502 14:06:14.374946 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:06:14.378134 15201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:14.378847 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:06:14.379109 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714658659912502-11301-0/minicluster-data/ts-0-root
uuid: "1f4a05af65de4ff880d46dc181844eb7"
format_stamp: "Formatted at 2024-05-02 14:06:14 on dist-test-slave-j72s"
I20240502 14:06:14.379391 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714658659912502-11301-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714658659912502-11301-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714658659912502-11301-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:14.400086 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:14.401100 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:14.402503 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:14.404650 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:14.404832 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:14.405061 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:14.405220 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:14.445631 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.65:39651
I20240502 14:06:14.445701 15263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.65:39651 every 8 connection(s)
I20240502 14:06:14.450373 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:14.458051 15268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:14.459988 15269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:14.461608 11301 server_base.cc:1034] running on GCE node
W20240502 14:06:14.462213 15272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:14.462827 15264 heartbeater.cc:347] Connected to a master server at 127.11.9.126:35053
I20240502 14:06:14.463157 15264 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:14.463174 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:06:14.463536 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:14.463680 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658774463660 us; error 0 us; skew 500 ppm
I20240502 14:06:14.464020 15264 heartbeater.cc:511] Master 127.11.9.126:35053 requested a full tablet report, sending...
I20240502 14:06:14.464255 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:14.466058 15137 ts_manager.cc:194] Registered new tserver with Master: 1f4a05af65de4ff880d46dc181844eb7 (127.11.9.65:39651)
I20240502 14:06:14.467664 15137 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45708
I20240502 14:06:14.468842 11301 webserver.cc:460] Webserver started at http://127.11.9.66:37527/ using document root <none> and password file <none>
I20240502 14:06:14.469463 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:14.469741 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:14.470072 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:14.471153 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714658659912502-11301-0/minicluster-data/ts-1-root/instance:
uuid: "52e5bfe7205e4d1f82a5cd5fa58451ab"
format_stamp: "Formatted at 2024-05-02 14:06:14 on dist-test-slave-j72s"
I20240502 14:06:14.475390 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:06:14.478314 15276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:14.479074 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:06:14.479358 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714658659912502-11301-0/minicluster-data/ts-1-root
uuid: "52e5bfe7205e4d1f82a5cd5fa58451ab"
format_stamp: "Formatted at 2024-05-02 14:06:14 on dist-test-slave-j72s"
I20240502 14:06:14.479595 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714658659912502-11301-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714658659912502-11301-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714658659912502-11301-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:14.487987 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:14.488991 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:14.490377 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:14.492429 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:14.492612 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:14.492827 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:14.492972 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:14.530397 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.66:37507
I20240502 14:06:14.530501 15338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.66:37507 every 8 connection(s)
I20240502 14:06:14.534897 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:14.541635 15342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:14.543773 15343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:14.545861 15345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:14.546541 11301 server_base.cc:1034] running on GCE node
I20240502 14:06:14.547403 15339 heartbeater.cc:347] Connected to a master server at 127.11.9.126:35053
I20240502 14:06:14.547436 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 14:06:14.547870 15339 heartbeater.cc:464] Registering TS with master...
W20240502 14:06:14.547859 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:14.548341 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658774548318 us; error 0 us; skew 500 ppm
I20240502 14:06:14.548730 15339 heartbeater.cc:511] Master 127.11.9.126:35053 requested a full tablet report, sending...
I20240502 14:06:14.548892 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:14.550721 15137 ts_manager.cc:194] Registered new tserver with Master: 52e5bfe7205e4d1f82a5cd5fa58451ab (127.11.9.66:37507)
I20240502 14:06:14.551393 11301 webserver.cc:460] Webserver started at http://127.11.9.67:41643/ using document root <none> and password file <none>
I20240502 14:06:14.551826 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:14.551995 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:14.552248 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:14.552167 15137 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45710
I20240502 14:06:14.553601 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714658659912502-11301-0/minicluster-data/ts-2-root/instance:
uuid: "f039467bfae0422f962a3411fcd332fd"
format_stamp: "Formatted at 2024-05-02 14:06:14 on dist-test-slave-j72s"
I20240502 14:06:14.557667 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20240502 14:06:14.560535 15350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:14.561213 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:06:14.561470 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714658659912502-11301-0/minicluster-data/ts-2-root
uuid: "f039467bfae0422f962a3411fcd332fd"
format_stamp: "Formatted at 2024-05-02 14:06:14 on dist-test-slave-j72s"
I20240502 14:06:14.561767 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714658659912502-11301-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714658659912502-11301-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714658659912502-11301-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:14.589738 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:14.590751 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:14.592098 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:14.594308 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:14.594501 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:14.594712 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:14.594866 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:14.631590 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.67:33141
I20240502 14:06:14.631685 15412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.67:33141 every 8 connection(s)
I20240502 14:06:14.644469 15413 heartbeater.cc:347] Connected to a master server at 127.11.9.126:35053
I20240502 14:06:14.644816 15413 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:14.645488 15413 heartbeater.cc:511] Master 127.11.9.126:35053 requested a full tablet report, sending...
I20240502 14:06:14.647234 15137 ts_manager.cc:194] Registered new tserver with Master: f039467bfae0422f962a3411fcd332fd (127.11.9.67:33141)
I20240502 14:06:14.647655 11301 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012978087s
I20240502 14:06:14.648536 15137 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43260
I20240502 14:06:15.470420 15264 heartbeater.cc:503] Master 127.11.9.126:35053 was elected leader, sending a full tablet report...
I20240502 14:06:15.554468 15339 heartbeater.cc:503] Master 127.11.9.126:35053 was elected leader, sending a full tablet report...
I20240502 14:06:15.651126 15413 heartbeater.cc:503] Master 127.11.9.126:35053 was elected leader, sending a full tablet report...
I20240502 14:06:15.679610 11301 test_util.cc:268] Using random seed: 456933988
I20240502 14:06:15.701429 15137 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43276:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:06:15.703337 15137 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:06:15.741016 15304 tablet_service.cc:1467] Processing CreateTablet for tablet 672e4068264644e5ae4959787268cdde (DEFAULT_TABLE table=test-workload [id=78df25e80654417591f4800da7bf8c68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:06:15.742918 15304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 672e4068264644e5ae4959787268cdde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:15.744833 15378 tablet_service.cc:1467] Processing CreateTablet for tablet 672e4068264644e5ae4959787268cdde (DEFAULT_TABLE table=test-workload [id=78df25e80654417591f4800da7bf8c68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:06:15.746083 15378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 672e4068264644e5ae4959787268cdde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:15.752069 15229 tablet_service.cc:1467] Processing CreateTablet for tablet 672e4068264644e5ae4959787268cdde (DEFAULT_TABLE table=test-workload [id=78df25e80654417591f4800da7bf8c68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:06:15.752950 15229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 672e4068264644e5ae4959787268cdde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:15.760005 15433 tablet_bootstrap.cc:492] T 672e4068264644e5ae4959787268cdde P 52e5bfe7205e4d1f82a5cd5fa58451ab: Bootstrap starting.
I20240502 14:06:15.762516 15434 tablet_bootstrap.cc:492] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd: Bootstrap starting.
I20240502 14:06:15.767439 15435 tablet_bootstrap.cc:492] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7: Bootstrap starting.
I20240502 14:06:15.767683 15433 tablet_bootstrap.cc:654] T 672e4068264644e5ae4959787268cdde P 52e5bfe7205e4d1f82a5cd5fa58451ab: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:15.769548 15434 tablet_bootstrap.cc:654] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:15.772219 15433 tablet_bootstrap.cc:492] T 672e4068264644e5ae4959787268cdde P 52e5bfe7205e4d1f82a5cd5fa58451ab: No bootstrap required, opened a new log
I20240502 14:06:15.772701 15433 ts_tablet_manager.cc:1367] T 672e4068264644e5ae4959787268cdde P 52e5bfe7205e4d1f82a5cd5fa58451ab: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20240502 14:06:15.773613 15435 tablet_bootstrap.cc:654] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:15.774981 15433 raft_consensus.cc:348] T 672e4068264644e5ae4959787268cdde P 52e5bfe7205e4d1f82a5cd5fa58451ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e5bfe7205e4d1f82a5cd5fa58451ab" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 37507 } } peers { permanent_uuid: "f039467bfae0422f962a3411fcd332fd" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 33141 } } peers { permanent_uuid: "1f4a05af65de4ff880d46dc181844eb7" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 39651 } }
I20240502 14:06:15.775537 15433 raft_consensus.cc:374] T 672e4068264644e5ae4959787268cdde P 52e5bfe7205e4d1f82a5cd5fa58451ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:15.775768 15433 raft_consensus.cc:729] T 672e4068264644e5ae4959787268cdde P 52e5bfe7205e4d1f82a5cd5fa58451ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52e5bfe7205e4d1f82a5cd5fa58451ab, State: Initialized, Role: FOLLOWER
I20240502 14:06:15.776355 15433 consensus_queue.cc:260] T 672e4068264644e5ae4959787268cdde P 52e5bfe7205e4d1f82a5cd5fa58451ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e5bfe7205e4d1f82a5cd5fa58451ab" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 37507 } } peers { permanent_uuid: "f039467bfae0422f962a3411fcd332fd" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 33141 } } peers { permanent_uuid: "1f4a05af65de4ff880d46dc181844eb7" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 39651 } }
I20240502 14:06:15.778862 15433 ts_tablet_manager.cc:1398] T 672e4068264644e5ae4959787268cdde P 52e5bfe7205e4d1f82a5cd5fa58451ab: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240502 14:06:15.780092 15435 tablet_bootstrap.cc:492] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7: No bootstrap required, opened a new log
I20240502 14:06:15.780656 15435 ts_tablet_manager.cc:1367] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20240502 14:06:15.783563 15434 tablet_bootstrap.cc:492] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd: No bootstrap required, opened a new log
I20240502 14:06:15.783391 15435 raft_consensus.cc:348] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e5bfe7205e4d1f82a5cd5fa58451ab" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 37507 } } peers { permanent_uuid: "f039467bfae0422f962a3411fcd332fd" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 33141 } } peers { permanent_uuid: "1f4a05af65de4ff880d46dc181844eb7" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 39651 } }
I20240502 14:06:15.783957 15434 ts_tablet_manager.cc:1367] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.004s
I20240502 14:06:15.784251 15435 raft_consensus.cc:374] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:15.784698 15435 raft_consensus.cc:729] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f4a05af65de4ff880d46dc181844eb7, State: Initialized, Role: FOLLOWER
W20240502 14:06:15.785177 15340 tablet.cc:2369] T 672e4068264644e5ae4959787268cdde P 52e5bfe7205e4d1f82a5cd5fa58451ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:06:15.785557 15435 consensus_queue.cc:260] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e5bfe7205e4d1f82a5cd5fa58451ab" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 37507 } } peers { permanent_uuid: "f039467bfae0422f962a3411fcd332fd" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 33141 } } peers { permanent_uuid: "1f4a05af65de4ff880d46dc181844eb7" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 39651 } }
I20240502 14:06:15.786662 15434 raft_consensus.cc:348] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e5bfe7205e4d1f82a5cd5fa58451ab" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 37507 } } peers { permanent_uuid: "f039467bfae0422f962a3411fcd332fd" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 33141 } } peers { permanent_uuid: "1f4a05af65de4ff880d46dc181844eb7" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 39651 } }
I20240502 14:06:15.787297 15434 raft_consensus.cc:374] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:15.787650 15434 raft_consensus.cc:729] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f039467bfae0422f962a3411fcd332fd, State: Initialized, Role: FOLLOWER
I20240502 14:06:15.788606 15435 ts_tablet_manager.cc:1398] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240502 14:06:15.788486 15434 consensus_queue.cc:260] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e5bfe7205e4d1f82a5cd5fa58451ab" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 37507 } } peers { permanent_uuid: "f039467bfae0422f962a3411fcd332fd" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 33141 } } peers { permanent_uuid: "1f4a05af65de4ff880d46dc181844eb7" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 39651 } }
I20240502 14:06:15.792064 15434 ts_tablet_manager.cc:1398] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20240502 14:06:15.850729 15440 raft_consensus.cc:482] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:15.851192 15440 raft_consensus.cc:504] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e5bfe7205e4d1f82a5cd5fa58451ab" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 37507 } } peers { permanent_uuid: "f039467bfae0422f962a3411fcd332fd" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 33141 } } peers { permanent_uuid: "1f4a05af65de4ff880d46dc181844eb7" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 39651 } }
I20240502 14:06:15.853073 15440 leader_election.cc:288] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52e5bfe7205e4d1f82a5cd5fa58451ab (127.11.9.66:37507), f039467bfae0422f962a3411fcd332fd (127.11.9.67:33141)
I20240502 14:06:15.863145 15388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672e4068264644e5ae4959787268cdde" candidate_uuid: "1f4a05af65de4ff880d46dc181844eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f039467bfae0422f962a3411fcd332fd" is_pre_election: true
I20240502 14:06:15.863344 15314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672e4068264644e5ae4959787268cdde" candidate_uuid: "1f4a05af65de4ff880d46dc181844eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52e5bfe7205e4d1f82a5cd5fa58451ab" is_pre_election: true
I20240502 14:06:15.863801 15388 raft_consensus.cc:2454] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f4a05af65de4ff880d46dc181844eb7 in term 0.
I20240502 14:06:15.863974 15314 raft_consensus.cc:2454] T 672e4068264644e5ae4959787268cdde P 52e5bfe7205e4d1f82a5cd5fa58451ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f4a05af65de4ff880d46dc181844eb7 in term 0.
I20240502 14:06:15.864785 15204 leader_election.cc:302] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f4a05af65de4ff880d46dc181844eb7, f039467bfae0422f962a3411fcd332fd; no voters:
I20240502 14:06:15.865531 15440 raft_consensus.cc:2781] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:15.865797 15440 raft_consensus.cc:482] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:15.865998 15440 raft_consensus.cc:3037] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:15.870896 15440 raft_consensus.cc:504] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e5bfe7205e4d1f82a5cd5fa58451ab" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 37507 } } peers { permanent_uuid: "f039467bfae0422f962a3411fcd332fd" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 33141 } } peers { permanent_uuid: "1f4a05af65de4ff880d46dc181844eb7" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 39651 } }
I20240502 14:06:15.872273 15440 leader_election.cc:288] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7 [CANDIDATE]: Term 1 election: Requested vote from peers 52e5bfe7205e4d1f82a5cd5fa58451ab (127.11.9.66:37507), f039467bfae0422f962a3411fcd332fd (127.11.9.67:33141)
I20240502 14:06:15.873024 15314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672e4068264644e5ae4959787268cdde" candidate_uuid: "1f4a05af65de4ff880d46dc181844eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52e5bfe7205e4d1f82a5cd5fa58451ab"
I20240502 14:06:15.873140 15388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672e4068264644e5ae4959787268cdde" candidate_uuid: "1f4a05af65de4ff880d46dc181844eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f039467bfae0422f962a3411fcd332fd"
I20240502 14:06:15.873500 15314 raft_consensus.cc:3037] T 672e4068264644e5ae4959787268cdde P 52e5bfe7205e4d1f82a5cd5fa58451ab [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:15.873646 15388 raft_consensus.cc:3037] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:15.878132 15314 raft_consensus.cc:2454] T 672e4068264644e5ae4959787268cdde P 52e5bfe7205e4d1f82a5cd5fa58451ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f4a05af65de4ff880d46dc181844eb7 in term 1.
I20240502 14:06:15.878252 15388 raft_consensus.cc:2454] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f4a05af65de4ff880d46dc181844eb7 in term 1.
I20240502 14:06:15.879006 15203 leader_election.cc:302] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f4a05af65de4ff880d46dc181844eb7, 52e5bfe7205e4d1f82a5cd5fa58451ab; no voters:
I20240502 14:06:15.879745 15440 raft_consensus.cc:2781] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:15.880841 15440 raft_consensus.cc:686] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7 [term 1 LEADER]: Becoming Leader. State: Replica: 1f4a05af65de4ff880d46dc181844eb7, State: Running, Role: LEADER
I20240502 14:06:15.881543 15440 consensus_queue.cc:237] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e5bfe7205e4d1f82a5cd5fa58451ab" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 37507 } } peers { permanent_uuid: "f039467bfae0422f962a3411fcd332fd" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 33141 } } peers { permanent_uuid: "1f4a05af65de4ff880d46dc181844eb7" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 39651 } }
I20240502 14:06:15.888334 15137 catalog_manager.cc:5484] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f4a05af65de4ff880d46dc181844eb7 (127.11.9.65). New cstate: current_term: 1 leader_uuid: "1f4a05af65de4ff880d46dc181844eb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e5bfe7205e4d1f82a5cd5fa58451ab" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 37507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f039467bfae0422f962a3411fcd332fd" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 33141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f4a05af65de4ff880d46dc181844eb7" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 39651 } health_report { overall_health: HEALTHY } } }
I20240502 14:06:15.900077 11301 tablet_server.cc:178] TabletServer@127.11.9.65:0 shutting down...
I20240502 14:06:15.919673 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:06:15.920390 11301 tablet_replica.cc:330] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7: stopping tablet replica
I20240502 14:06:15.920877 11301 raft_consensus.cc:2229] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:15.921428 11301 pending_rounds.cc:62] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7: Trying to abort 1 pending ops.
I20240502 14:06:15.921670 11301 pending_rounds.cc:68] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023242346017689600 op_type: NO_OP noop_request { }
I20240502 14:06:15.922024 11301 raft_consensus.cc:2866] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 14:06:15.922289 11301 raft_consensus.cc:2258] T 672e4068264644e5ae4959787268cdde P 1f4a05af65de4ff880d46dc181844eb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:15.931288 11301 tablet_server.cc:194] TabletServer@127.11.9.65:0 shutdown complete.
I20240502 14:06:16.288081 15190 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:06:16.289350 15190 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:06:16.289685 15190 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:06:17.483227 15446 raft_consensus.cc:482] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:17.483716 15446 raft_consensus.cc:504] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e5bfe7205e4d1f82a5cd5fa58451ab" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 37507 } } peers { permanent_uuid: "f039467bfae0422f962a3411fcd332fd" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 33141 } } peers { permanent_uuid: "1f4a05af65de4ff880d46dc181844eb7" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 39651 } }
I20240502 14:06:17.485499 15446 leader_election.cc:288] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52e5bfe7205e4d1f82a5cd5fa58451ab (127.11.9.66:37507), 1f4a05af65de4ff880d46dc181844eb7 (127.11.9.65:39651)
W20240502 14:06:17.492002 15353 leader_election.cc:334] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1f4a05af65de4ff880d46dc181844eb7 (127.11.9.65:39651): Network error: Client connection negotiation failed: client connection to 127.11.9.65:39651: connect: Connection refused (error 111)
I20240502 14:06:17.495085 15314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672e4068264644e5ae4959787268cdde" candidate_uuid: "f039467bfae0422f962a3411fcd332fd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52e5bfe7205e4d1f82a5cd5fa58451ab" is_pre_election: true
I20240502 14:06:17.495723 15314 raft_consensus.cc:2454] T 672e4068264644e5ae4959787268cdde P 52e5bfe7205e4d1f82a5cd5fa58451ab [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f039467bfae0422f962a3411fcd332fd in term 1.
I20240502 14:06:17.496603 15352 leader_election.cc:302] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52e5bfe7205e4d1f82a5cd5fa58451ab, f039467bfae0422f962a3411fcd332fd; no voters: 1f4a05af65de4ff880d46dc181844eb7
I20240502 14:06:17.497229 15446 raft_consensus.cc:2781] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:06:17.497473 15446 raft_consensus.cc:482] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:17.497753 15446 raft_consensus.cc:3037] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:17.502372 15446 raft_consensus.cc:504] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e5bfe7205e4d1f82a5cd5fa58451ab" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 37507 } } peers { permanent_uuid: "f039467bfae0422f962a3411fcd332fd" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 33141 } } peers { permanent_uuid: "1f4a05af65de4ff880d46dc181844eb7" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 39651 } }
I20240502 14:06:17.503703 15446 leader_election.cc:288] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd [CANDIDATE]: Term 2 election: Requested vote from peers 52e5bfe7205e4d1f82a5cd5fa58451ab (127.11.9.66:37507), 1f4a05af65de4ff880d46dc181844eb7 (127.11.9.65:39651)
I20240502 14:06:17.504472 15314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672e4068264644e5ae4959787268cdde" candidate_uuid: "f039467bfae0422f962a3411fcd332fd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52e5bfe7205e4d1f82a5cd5fa58451ab"
I20240502 14:06:17.504918 15314 raft_consensus.cc:3037] T 672e4068264644e5ae4959787268cdde P 52e5bfe7205e4d1f82a5cd5fa58451ab [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:06:17.506939 15353 leader_election.cc:334] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1f4a05af65de4ff880d46dc181844eb7 (127.11.9.65:39651): Network error: Client connection negotiation failed: client connection to 127.11.9.65:39651: connect: Connection refused (error 111)
I20240502 14:06:17.509214 15314 raft_consensus.cc:2454] T 672e4068264644e5ae4959787268cdde P 52e5bfe7205e4d1f82a5cd5fa58451ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f039467bfae0422f962a3411fcd332fd in term 2.
I20240502 14:06:17.510054 15352 leader_election.cc:302] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52e5bfe7205e4d1f82a5cd5fa58451ab, f039467bfae0422f962a3411fcd332fd; no voters: 1f4a05af65de4ff880d46dc181844eb7
I20240502 14:06:17.510612 15446 raft_consensus.cc:2781] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:06:17.511539 15446 raft_consensus.cc:686] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd [term 2 LEADER]: Becoming Leader. State: Replica: f039467bfae0422f962a3411fcd332fd, State: Running, Role: LEADER
I20240502 14:06:17.512152 15446 consensus_queue.cc:237] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e5bfe7205e4d1f82a5cd5fa58451ab" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 37507 } } peers { permanent_uuid: "f039467bfae0422f962a3411fcd332fd" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 33141 } } peers { permanent_uuid: "1f4a05af65de4ff880d46dc181844eb7" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 39651 } }
I20240502 14:06:17.518149 15137 catalog_manager.cc:5484] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd reported cstate change: term changed from 1 to 2, leader changed from 1f4a05af65de4ff880d46dc181844eb7 (127.11.9.65) to f039467bfae0422f962a3411fcd332fd (127.11.9.67). New cstate: current_term: 2 leader_uuid: "f039467bfae0422f962a3411fcd332fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52e5bfe7205e4d1f82a5cd5fa58451ab" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 37507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f039467bfae0422f962a3411fcd332fd" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 33141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f4a05af65de4ff880d46dc181844eb7" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 39651 } health_report { overall_health: UNKNOWN } } }
W20240502 14:06:17.926659 15353 consensus_peers.cc:485] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd -> Peer 1f4a05af65de4ff880d46dc181844eb7 (127.11.9.65:39651): Couldn't send request to peer 1f4a05af65de4ff880d46dc181844eb7. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.65:39651: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:17.992609 15314 raft_consensus.cc:1261] T 672e4068264644e5ae4959787268cdde P 52e5bfe7205e4d1f82a5cd5fa58451ab [term 2 FOLLOWER]: Refusing update from remote peer f039467bfae0422f962a3411fcd332fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 14:06:17.993938 15446 consensus_queue.cc:1035] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "52e5bfe7205e4d1f82a5cd5fa58451ab" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 37507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:18.290611 15190 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:06:18.292068 15190 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:06:18.292474 15190 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 14:06:18.401011 15353 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.9.65:39651: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240502 14:06:20.293380 15190 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:06:20.294677 15190 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:06:20.294955 15190 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 14:06:20.481902 15353 consensus_peers.cc:485] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd -> Peer 1f4a05af65de4ff880d46dc181844eb7 (127.11.9.65:39651): Couldn't send request to peer 1f4a05af65de4ff880d46dc181844eb7. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.65:39651: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:06:22.295974 15190 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:06:22.297283 15190 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:06:22.297590 15190 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 14:06:23.077539 15353 consensus_peers.cc:485] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd -> Peer 1f4a05af65de4ff880d46dc181844eb7 (127.11.9.65:39651): Couldn't send request to peer 1f4a05af65de4ff880d46dc181844eb7. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.65:39651: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:06:23.616374 15353 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.9.65:39651: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 14:06:24.336941 15190 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:06:24.338850 15190 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:06:24.339241 15190 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240502 14:06:25.333233 15335 debug-util.cc:398] Leaking SignalData structure 0x7b08000e23e0 after lost signal to thread 11304
W20240502 14:06:25.334170 15335 debug-util.cc:398] Leaking SignalData structure 0x7b08000e93a0 after lost signal to thread 15171
W20240502 14:06:25.335093 15335 debug-util.cc:398] Leaking SignalData structure 0x7b08001c5140 after lost signal to thread 15338
W20240502 14:06:25.335937 15335 debug-util.cc:398] Leaking SignalData structure 0x7b08001002e0 after lost signal to thread 15412
W20240502 14:06:25.573590 15353 consensus_peers.cc:485] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd -> Peer 1f4a05af65de4ff880d46dc181844eb7 (127.11.9.65:39651): Couldn't send request to peer 1f4a05af65de4ff880d46dc181844eb7. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.65:39651: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:06:26.339941 15190 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 14:06:26.341070 15190 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 14:06:26.341359 15190 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 14:06:27.098723 11301 tablet_server.cc:178] TabletServer@127.11.9.66:0 shutting down...
I20240502 14:06:27.129032 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:06:27.129767 11301 tablet_replica.cc:330] T 672e4068264644e5ae4959787268cdde P 52e5bfe7205e4d1f82a5cd5fa58451ab: stopping tablet replica
I20240502 14:06:27.130487 11301 raft_consensus.cc:2229] T 672e4068264644e5ae4959787268cdde P 52e5bfe7205e4d1f82a5cd5fa58451ab [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:27.131220 11301 raft_consensus.cc:2258] T 672e4068264644e5ae4959787268cdde P 52e5bfe7205e4d1f82a5cd5fa58451ab [term 2 FOLLOWER]: Raft consensus is shut down!
W20240502 14:06:27.272294 15352 consensus_peers.cc:485] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd -> Peer 52e5bfe7205e4d1f82a5cd5fa58451ab (127.11.9.66:37507): Couldn't send request to peer 52e5bfe7205e4d1f82a5cd5fa58451ab. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.66:37507: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:27.770241 11301 tablet_server.cc:194] TabletServer@127.11.9.66:0 shutdown complete.
I20240502 14:06:27.780689 11301 tablet_server.cc:178] TabletServer@127.11.9.67:0 shutting down...
I20240502 14:06:27.800495 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:06:27.801096 11301 tablet_replica.cc:330] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd: stopping tablet replica
I20240502 14:06:27.801652 11301 raft_consensus.cc:2229] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd [term 2 LEADER]: Raft consensus shutting down.
I20240502 14:06:27.802503 11301 raft_consensus.cc:2258] T 672e4068264644e5ae4959787268cdde P f039467bfae0422f962a3411fcd332fd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:27.821419 11301 tablet_server.cc:194] TabletServer@127.11.9.67:0 shutdown complete.
I20240502 14:06:27.831617 11301 master.cc:537] Master@127.11.9.126:35053 shutting down...
I20240502 14:06:27.850286 11301 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8157ceecdb1348259d010953425dfc43 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:27.850894 11301 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8157ceecdb1348259d010953425dfc43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:27.851208 11301 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8157ceecdb1348259d010953425dfc43: stopping tablet replica
I20240502 14:06:27.869277 11301 master.cc:552] Master@127.11.9.126:35053 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor (13746 ms)
[ RUN ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering
I20240502 14:06:27.906802 11301 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.9.126:40531
I20240502 14:06:27.907907 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:27.913761 15482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:27.914392 15483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:27.914880 15485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:27.915576 11301 server_base.cc:1034] running on GCE node
I20240502 14:06:27.916393 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:06:27.916568 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:27.916676 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658787916666 us; error 0 us; skew 500 ppm
I20240502 14:06:27.917076 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:27.919385 11301 webserver.cc:460] Webserver started at http://127.11.9.126:33851/ using document root <none> and password file <none>
I20240502 14:06:27.919802 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:27.919952 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:27.920171 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:27.921306 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714658659912502-11301-0/minicluster-data/master-0-root/instance:
uuid: "2b72fc549a36485daec3609edf942b24"
format_stamp: "Formatted at 2024-05-02 14:06:27 on dist-test-slave-j72s"
I20240502 14:06:27.925441 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 14:06:27.928550 15490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:27.929191 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:06:27.929456 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714658659912502-11301-0/minicluster-data/master-0-root
uuid: "2b72fc549a36485daec3609edf942b24"
format_stamp: "Formatted at 2024-05-02 14:06:27 on dist-test-slave-j72s"
I20240502 14:06:27.929750 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714658659912502-11301-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714658659912502-11301-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714658659912502-11301-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:27.959158 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:27.960345 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:28.000761 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.126:40531
I20240502 14:06:28.000867 15541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.126:40531 every 8 connection(s)
I20240502 14:06:28.004498 15542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:28.014848 15542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b72fc549a36485daec3609edf942b24: Bootstrap starting.
I20240502 14:06:28.019014 15542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b72fc549a36485daec3609edf942b24: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:28.022943 15542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b72fc549a36485daec3609edf942b24: No bootstrap required, opened a new log
I20240502 14:06:28.024806 15542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b72fc549a36485daec3609edf942b24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b72fc549a36485daec3609edf942b24" member_type: VOTER }
I20240502 14:06:28.025298 15542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b72fc549a36485daec3609edf942b24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:28.025542 15542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b72fc549a36485daec3609edf942b24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b72fc549a36485daec3609edf942b24, State: Initialized, Role: FOLLOWER
I20240502 14:06:28.026046 15542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b72fc549a36485daec3609edf942b24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b72fc549a36485daec3609edf942b24" member_type: VOTER }
I20240502 14:06:28.026506 15542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b72fc549a36485daec3609edf942b24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:06:28.026710 15542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b72fc549a36485daec3609edf942b24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:06:28.026980 15542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b72fc549a36485daec3609edf942b24 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:28.031330 15542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b72fc549a36485daec3609edf942b24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b72fc549a36485daec3609edf942b24" member_type: VOTER }
I20240502 14:06:28.031808 15542 leader_election.cc:302] T 00000000000000000000000000000000 P 2b72fc549a36485daec3609edf942b24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b72fc549a36485daec3609edf942b24; no voters:
I20240502 14:06:28.032872 15542 leader_election.cc:288] T 00000000000000000000000000000000 P 2b72fc549a36485daec3609edf942b24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:06:28.033231 15545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b72fc549a36485daec3609edf942b24 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:28.034497 15545 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b72fc549a36485daec3609edf942b24 [term 1 LEADER]: Becoming Leader. State: Replica: 2b72fc549a36485daec3609edf942b24, State: Running, Role: LEADER
I20240502 14:06:28.035188 15545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b72fc549a36485daec3609edf942b24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b72fc549a36485daec3609edf942b24" member_type: VOTER }
I20240502 14:06:28.035701 15542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b72fc549a36485daec3609edf942b24 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:06:28.039849 15547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b72fc549a36485daec3609edf942b24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b72fc549a36485daec3609edf942b24. Latest consensus state: current_term: 1 leader_uuid: "2b72fc549a36485daec3609edf942b24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b72fc549a36485daec3609edf942b24" member_type: VOTER } }
I20240502 14:06:28.040566 15547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b72fc549a36485daec3609edf942b24 [sys.catalog]: This master's current role is: LEADER
I20240502 14:06:28.040589 15546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b72fc549a36485daec3609edf942b24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b72fc549a36485daec3609edf942b24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b72fc549a36485daec3609edf942b24" member_type: VOTER } }
I20240502 14:06:28.044176 15546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b72fc549a36485daec3609edf942b24 [sys.catalog]: This master's current role is: LEADER
I20240502 14:06:28.049937 15554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:06:28.051182 11301 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:06:28.054731 15554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:06:28.062568 15554 catalog_manager.cc:1347] Generated new cluster ID: c9cedc1ac4ba402d850a70000a7c314f
I20240502 14:06:28.062836 15554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:06:28.077548 15554 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:06:28.078749 15554 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:06:28.094010 15554 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2b72fc549a36485daec3609edf942b24: Generated new TSK 0
I20240502 14:06:28.094581 15554 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:06:28.117766 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:28.124684 15563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:28.126150 15564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:28.127179 15566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:28.128180 11301 server_base.cc:1034] running on GCE node
I20240502 14:06:28.128924 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:06:28.129096 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:28.129215 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658788129204 us; error 0 us; skew 500 ppm
I20240502 14:06:28.129720 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:28.132042 11301 webserver.cc:460] Webserver started at http://127.11.9.65:35491/ using document root <none> and password file <none>
I20240502 14:06:28.132455 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:28.132606 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:28.132807 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:28.133877 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714658659912502-11301-0/minicluster-data/ts-0-root/instance:
uuid: "17ee9f2434c74a02ad9a59074002db04"
format_stamp: "Formatted at 2024-05-02 14:06:28 on dist-test-slave-j72s"
I20240502 14:06:28.138258 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:06:28.141187 15571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:28.141935 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:06:28.142196 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714658659912502-11301-0/minicluster-data/ts-0-root
uuid: "17ee9f2434c74a02ad9a59074002db04"
format_stamp: "Formatted at 2024-05-02 14:06:28 on dist-test-slave-j72s"
I20240502 14:06:28.142465 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714658659912502-11301-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714658659912502-11301-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714658659912502-11301-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:28.155184 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:28.156188 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:28.157445 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:28.159742 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:28.159914 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:28.160142 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:28.160300 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:28.198444 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.65:37005
I20240502 14:06:28.198554 15633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.65:37005 every 8 connection(s)
I20240502 14:06:28.202766 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:28.209369 15638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:28.211283 15639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:28.212846 15641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:28.213907 11301 server_base.cc:1034] running on GCE node
I20240502 14:06:28.215013 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:06:28.215276 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:28.215471 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658788215454 us; error 0 us; skew 500 ppm
I20240502 14:06:28.216164 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:28.216845 15634 heartbeater.cc:347] Connected to a master server at 127.11.9.126:40531
I20240502 14:06:28.217290 15634 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:28.218123 15634 heartbeater.cc:511] Master 127.11.9.126:40531 requested a full tablet report, sending...
I20240502 14:06:28.218817 11301 webserver.cc:460] Webserver started at http://127.11.9.66:35765/ using document root <none> and password file <none>
I20240502 14:06:28.219413 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:28.219641 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:28.219944 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:28.220391 15507 ts_manager.cc:194] Registered new tserver with Master: 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005)
I20240502 14:06:28.221494 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714658659912502-11301-0/minicluster-data/ts-1-root/instance:
uuid: "ab642c5bd4b14d39b87f105702c613e5"
format_stamp: "Formatted at 2024-05-02 14:06:28 on dist-test-slave-j72s"
I20240502 14:06:28.222738 15507 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34932
I20240502 14:06:28.226751 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 14:06:28.229640 15646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:28.230360 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:06:28.230600 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714658659912502-11301-0/minicluster-data/ts-1-root
uuid: "ab642c5bd4b14d39b87f105702c613e5"
format_stamp: "Formatted at 2024-05-02 14:06:28 on dist-test-slave-j72s"
I20240502 14:06:28.230842 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714658659912502-11301-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714658659912502-11301-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714658659912502-11301-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:28.241618 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:28.242748 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:28.244154 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:28.246301 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:28.246487 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:28.246654 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:28.246773 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:28.284375 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.66:33825
I20240502 14:06:28.284472 15708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.66:33825 every 8 connection(s)
I20240502 14:06:28.289005 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:28.295450 15712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:28.297930 15713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:28.299576 15715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:28.300325 11301 server_base.cc:1034] running on GCE node
I20240502 14:06:28.300326 15709 heartbeater.cc:347] Connected to a master server at 127.11.9.126:40531
I20240502 14:06:28.300999 15709 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:28.301352 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:06:28.301596 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:28.301738 15709 heartbeater.cc:511] Master 127.11.9.126:40531 requested a full tablet report, sending...
I20240502 14:06:28.301750 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658788301729 us; error 0 us; skew 500 ppm
I20240502 14:06:28.302559 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:28.303783 15507 ts_manager.cc:194] Registered new tserver with Master: ab642c5bd4b14d39b87f105702c613e5 (127.11.9.66:33825)
I20240502 14:06:28.305088 11301 webserver.cc:460] Webserver started at http://127.11.9.67:44015/ using document root <none> and password file <none>
I20240502 14:06:28.305342 15507 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34946
I20240502 14:06:28.305585 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:28.305821 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:28.306098 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:28.307281 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714658659912502-11301-0/minicluster-data/ts-2-root/instance:
uuid: "f047b33669e648578b35878837bb0494"
format_stamp: "Formatted at 2024-05-02 14:06:28 on dist-test-slave-j72s"
I20240502 14:06:28.311199 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 14:06:28.314116 15720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:28.314749 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:06:28.315017 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714658659912502-11301-0/minicluster-data/ts-2-root
uuid: "f047b33669e648578b35878837bb0494"
format_stamp: "Formatted at 2024-05-02 14:06:28 on dist-test-slave-j72s"
I20240502 14:06:28.315251 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714658659912502-11301-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714658659912502-11301-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714658659912502-11301-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:28.329726 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:28.330750 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:28.332060 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:28.334137 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:28.334328 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:28.334535 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:28.334678 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:28.372655 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.67:38309
I20240502 14:06:28.372758 15782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.67:38309 every 8 connection(s)
I20240502 14:06:28.385321 15783 heartbeater.cc:347] Connected to a master server at 127.11.9.126:40531
I20240502 14:06:28.385691 15783 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:28.386370 15783 heartbeater.cc:511] Master 127.11.9.126:40531 requested a full tablet report, sending...
I20240502 14:06:28.388026 15507 ts_manager.cc:194] Registered new tserver with Master: f047b33669e648578b35878837bb0494 (127.11.9.67:38309)
I20240502 14:06:28.389096 11301 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013308876s
I20240502 14:06:28.389259 15507 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34958
I20240502 14:06:29.225541 15634 heartbeater.cc:503] Master 127.11.9.126:40531 was elected leader, sending a full tablet report...
I20240502 14:06:29.307510 15709 heartbeater.cc:503] Master 127.11.9.126:40531 was elected leader, sending a full tablet report...
I20240502 14:06:29.391528 15783 heartbeater.cc:503] Master 127.11.9.126:40531 was elected leader, sending a full tablet report...
I20240502 14:06:29.421036 11301 test_util.cc:268] Using random seed: 470675413
I20240502 14:06:29.441685 15507 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34962:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:06:29.443961 15507 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:06:29.498365 15748 tablet_service.cc:1467] Processing CreateTablet for tablet c808defc7a8b4ecda4490809db5f983b (DEFAULT_TABLE table=test-workload [id=b7e312dadc20472d9c58741722e9b268]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 14:06:29.499851 15748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c808defc7a8b4ecda4490809db5f983b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:29.500469 15599 tablet_service.cc:1467] Processing CreateTablet for tablet c808defc7a8b4ecda4490809db5f983b (DEFAULT_TABLE table=test-workload [id=b7e312dadc20472d9c58741722e9b268]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 14:06:29.501142 15598 tablet_service.cc:1467] Processing CreateTablet for tablet 56ac0dbb1eef484bba6413e33b6729b3 (DEFAULT_TABLE table=test-workload [id=b7e312dadc20472d9c58741722e9b268]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 14:06:29.501811 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c808defc7a8b4ecda4490809db5f983b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:29.501976 15596 tablet_service.cc:1467] Processing CreateTablet for tablet 8a727b7623ba4587bb183d2ce121ceed (DEFAULT_TABLE table=test-workload [id=b7e312dadc20472d9c58741722e9b268]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 14:06:29.502498 15598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ac0dbb1eef484bba6413e33b6729b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:29.502663 15747 tablet_service.cc:1467] Processing CreateTablet for tablet 56ac0dbb1eef484bba6413e33b6729b3 (DEFAULT_TABLE table=test-workload [id=b7e312dadc20472d9c58741722e9b268]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 14:06:29.503300 15596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a727b7623ba4587bb183d2ce121ceed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:29.504041 15747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ac0dbb1eef484bba6413e33b6729b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:29.504647 15597 tablet_service.cc:1467] Processing CreateTablet for tablet dd54f15c1b87413c8541c924088c1cde (DEFAULT_TABLE table=test-workload [id=b7e312dadc20472d9c58741722e9b268]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 14:06:29.505939 15597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd54f15c1b87413c8541c924088c1cde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:29.506762 15746 tablet_service.cc:1467] Processing CreateTablet for tablet dd54f15c1b87413c8541c924088c1cde (DEFAULT_TABLE table=test-workload [id=b7e312dadc20472d9c58741722e9b268]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 14:06:29.508026 15746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd54f15c1b87413c8541c924088c1cde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:29.511123 15745 tablet_service.cc:1467] Processing CreateTablet for tablet 8a727b7623ba4587bb183d2ce121ceed (DEFAULT_TABLE table=test-workload [id=b7e312dadc20472d9c58741722e9b268]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 14:06:29.512351 15745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a727b7623ba4587bb183d2ce121ceed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:29.534797 15673 tablet_service.cc:1467] Processing CreateTablet for tablet 56ac0dbb1eef484bba6413e33b6729b3 (DEFAULT_TABLE table=test-workload [id=b7e312dadc20472d9c58741722e9b268]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 14:06:29.536181 15673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ac0dbb1eef484bba6413e33b6729b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:29.536641 15671 tablet_service.cc:1467] Processing CreateTablet for tablet 8a727b7623ba4587bb183d2ce121ceed (DEFAULT_TABLE table=test-workload [id=b7e312dadc20472d9c58741722e9b268]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 14:06:29.537892 15671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a727b7623ba4587bb183d2ce121ceed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:29.540045 15672 tablet_service.cc:1467] Processing CreateTablet for tablet dd54f15c1b87413c8541c924088c1cde (DEFAULT_TABLE table=test-workload [id=b7e312dadc20472d9c58741722e9b268]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 14:06:29.533003 15674 tablet_service.cc:1467] Processing CreateTablet for tablet c808defc7a8b4ecda4490809db5f983b (DEFAULT_TABLE table=test-workload [id=b7e312dadc20472d9c58741722e9b268]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 14:06:29.541399 15672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd54f15c1b87413c8541c924088c1cde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:29.545950 15674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c808defc7a8b4ecda4490809db5f983b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:29.550715 15804 tablet_bootstrap.cc:492] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494: Bootstrap starting.
I20240502 14:06:29.556821 15804 tablet_bootstrap.cc:654] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:29.566120 15803 tablet_bootstrap.cc:492] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04: Bootstrap starting.
I20240502 14:06:29.580502 15805 tablet_bootstrap.cc:492] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5: Bootstrap starting.
I20240502 14:06:29.580974 15804 tablet_bootstrap.cc:492] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494: No bootstrap required, opened a new log
I20240502 14:06:29.581439 15804 ts_tablet_manager.cc:1367] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.012s
I20240502 14:06:29.584504 15803 tablet_bootstrap.cc:654] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:29.584121 15804 raft_consensus.cc:348] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:29.584908 15804 raft_consensus.cc:374] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:29.585233 15804 raft_consensus.cc:729] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f047b33669e648578b35878837bb0494, State: Initialized, Role: FOLLOWER
I20240502 14:06:29.585992 15804 consensus_queue.cc:260] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:29.586738 15805 tablet_bootstrap.cc:654] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:29.591823 15804 ts_tablet_manager.cc:1398] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20240502 14:06:29.592103 15803 tablet_bootstrap.cc:492] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04: No bootstrap required, opened a new log
I20240502 14:06:29.592684 15803 ts_tablet_manager.cc:1367] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.004s
I20240502 14:06:29.593060 15804 tablet_bootstrap.cc:492] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494: Bootstrap starting.
I20240502 14:06:29.595574 15803 raft_consensus.cc:348] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:29.596388 15803 raft_consensus.cc:374] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:29.596736 15803 raft_consensus.cc:729] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17ee9f2434c74a02ad9a59074002db04, State: Initialized, Role: FOLLOWER
I20240502 14:06:29.597498 15803 consensus_queue.cc:260] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:29.599057 15805 tablet_bootstrap.cc:492] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5: No bootstrap required, opened a new log
I20240502 14:06:29.599532 15805 ts_tablet_manager.cc:1367] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.007s
I20240502 14:06:29.600008 15804 tablet_bootstrap.cc:654] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:29.602388 15805 raft_consensus.cc:348] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:29.603328 15805 raft_consensus.cc:374] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:29.603727 15805 raft_consensus.cc:729] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab642c5bd4b14d39b87f105702c613e5, State: Initialized, Role: FOLLOWER
I20240502 14:06:29.604704 15805 consensus_queue.cc:260] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:29.609629 15810 raft_consensus.cc:482] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:29.610459 15803 ts_tablet_manager.cc:1398] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04: Time spent starting tablet: real 0.017s user 0.009s sys 0.008s
I20240502 14:06:29.611542 15803 tablet_bootstrap.cc:492] T c808defc7a8b4ecda4490809db5f983b P 17ee9f2434c74a02ad9a59074002db04: Bootstrap starting.
I20240502 14:06:29.612567 15805 ts_tablet_manager.cc:1398] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5: Time spent starting tablet: real 0.013s user 0.007s sys 0.003s
I20240502 14:06:29.613806 15805 tablet_bootstrap.cc:492] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5: Bootstrap starting.
I20240502 14:06:29.618655 15803 tablet_bootstrap.cc:654] T c808defc7a8b4ecda4490809db5f983b P 17ee9f2434c74a02ad9a59074002db04: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:29.621145 15805 tablet_bootstrap.cc:654] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:29.610091 15810 raft_consensus.cc:504] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:29.630478 15803 tablet_bootstrap.cc:492] T c808defc7a8b4ecda4490809db5f983b P 17ee9f2434c74a02ad9a59074002db04: No bootstrap required, opened a new log
I20240502 14:06:29.630600 15804 tablet_bootstrap.cc:492] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494: No bootstrap required, opened a new log
I20240502 14:06:29.630980 15803 ts_tablet_manager.cc:1367] T c808defc7a8b4ecda4490809db5f983b P 17ee9f2434c74a02ad9a59074002db04: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.014s
I20240502 14:06:29.631323 15805 tablet_bootstrap.cc:492] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5: No bootstrap required, opened a new log
I20240502 14:06:29.631371 15804 ts_tablet_manager.cc:1367] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494: Time spent bootstrapping tablet: real 0.039s user 0.021s sys 0.015s
I20240502 14:06:29.631896 15805 ts_tablet_manager.cc:1367] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.008s
I20240502 14:06:29.633958 15803 raft_consensus.cc:348] T c808defc7a8b4ecda4490809db5f983b P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } }
I20240502 14:06:29.634644 15803 raft_consensus.cc:374] T c808defc7a8b4ecda4490809db5f983b P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:29.634368 15804 raft_consensus.cc:348] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:29.634944 15803 raft_consensus.cc:729] T c808defc7a8b4ecda4490809db5f983b P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17ee9f2434c74a02ad9a59074002db04, State: Initialized, Role: FOLLOWER
I20240502 14:06:29.635318 15804 raft_consensus.cc:374] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:29.635742 15804 raft_consensus.cc:729] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f047b33669e648578b35878837bb0494, State: Initialized, Role: FOLLOWER
I20240502 14:06:29.635710 15803 consensus_queue.cc:260] T c808defc7a8b4ecda4490809db5f983b P 17ee9f2434c74a02ad9a59074002db04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } }
I20240502 14:06:29.636960 15804 consensus_queue.cc:260] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:29.638001 15803 ts_tablet_manager.cc:1398] T c808defc7a8b4ecda4490809db5f983b P 17ee9f2434c74a02ad9a59074002db04: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 14:06:29.639161 15803 tablet_bootstrap.cc:492] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04: Bootstrap starting.
I20240502 14:06:29.639199 15804 ts_tablet_manager.cc:1398] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 14:06:29.640347 15804 tablet_bootstrap.cc:492] T 56ac0dbb1eef484bba6413e33b6729b3 P f047b33669e648578b35878837bb0494: Bootstrap starting.
I20240502 14:06:29.643267 15805 raft_consensus.cc:348] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } }
I20240502 14:06:29.643923 15805 raft_consensus.cc:374] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:29.644203 15805 raft_consensus.cc:729] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab642c5bd4b14d39b87f105702c613e5, State: Initialized, Role: FOLLOWER
I20240502 14:06:29.646255 15804 tablet_bootstrap.cc:654] T 56ac0dbb1eef484bba6413e33b6729b3 P f047b33669e648578b35878837bb0494: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:29.646387 15803 tablet_bootstrap.cc:654] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:29.658762 15810 leader_election.cc:288] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab642c5bd4b14d39b87f105702c613e5 (127.11.9.66:33825), f047b33669e648578b35878837bb0494 (127.11.9.67:38309)
I20240502 14:06:29.660645 15805 consensus_queue.cc:260] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } }
I20240502 14:06:29.663494 15805 ts_tablet_manager.cc:1398] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5: Time spent starting tablet: real 0.031s user 0.004s sys 0.004s
I20240502 14:06:29.673305 15805 tablet_bootstrap.cc:492] T 8a727b7623ba4587bb183d2ce121ceed P ab642c5bd4b14d39b87f105702c613e5: Bootstrap starting.
I20240502 14:06:29.680505 15805 tablet_bootstrap.cc:654] T 8a727b7623ba4587bb183d2ce121ceed P ab642c5bd4b14d39b87f105702c613e5: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:29.684026 15803 tablet_bootstrap.cc:492] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04: No bootstrap required, opened a new log
I20240502 14:06:29.684502 15803 ts_tablet_manager.cc:1367] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04: Time spent bootstrapping tablet: real 0.046s user 0.017s sys 0.026s
I20240502 14:06:29.687813 15803 raft_consensus.cc:348] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:29.688809 15803 raft_consensus.cc:374] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:29.689175 15803 raft_consensus.cc:729] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17ee9f2434c74a02ad9a59074002db04, State: Initialized, Role: FOLLOWER
I20240502 14:06:29.690078 15803 consensus_queue.cc:260] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:29.692478 15803 ts_tablet_manager.cc:1398] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240502 14:06:29.693420 15803 tablet_bootstrap.cc:492] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04: Bootstrap starting.
I20240502 14:06:29.695585 15805 tablet_bootstrap.cc:492] T 8a727b7623ba4587bb183d2ce121ceed P ab642c5bd4b14d39b87f105702c613e5: No bootstrap required, opened a new log
I20240502 14:06:29.696105 15805 ts_tablet_manager.cc:1367] T 8a727b7623ba4587bb183d2ce121ceed P ab642c5bd4b14d39b87f105702c613e5: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.007s
I20240502 14:06:29.699030 15803 tablet_bootstrap.cc:654] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:29.698935 15805 raft_consensus.cc:348] T 8a727b7623ba4587bb183d2ce121ceed P ab642c5bd4b14d39b87f105702c613e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:29.699775 15805 raft_consensus.cc:374] T 8a727b7623ba4587bb183d2ce121ceed P ab642c5bd4b14d39b87f105702c613e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:29.700093 15805 raft_consensus.cc:729] T 8a727b7623ba4587bb183d2ce121ceed P ab642c5bd4b14d39b87f105702c613e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab642c5bd4b14d39b87f105702c613e5, State: Initialized, Role: FOLLOWER
I20240502 14:06:29.701049 15805 consensus_queue.cc:260] T 8a727b7623ba4587bb183d2ce121ceed P ab642c5bd4b14d39b87f105702c613e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:29.710373 15805 ts_tablet_manager.cc:1398] T 8a727b7623ba4587bb183d2ce121ceed P ab642c5bd4b14d39b87f105702c613e5: Time spent starting tablet: real 0.014s user 0.008s sys 0.001s
I20240502 14:06:29.710724 15684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a727b7623ba4587bb183d2ce121ceed" candidate_uuid: "17ee9f2434c74a02ad9a59074002db04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab642c5bd4b14d39b87f105702c613e5" is_pre_election: true
I20240502 14:06:29.711346 15805 tablet_bootstrap.cc:492] T dd54f15c1b87413c8541c924088c1cde P ab642c5bd4b14d39b87f105702c613e5: Bootstrap starting.
I20240502 14:06:29.711480 15684 raft_consensus.cc:2454] T 8a727b7623ba4587bb183d2ce121ceed P ab642c5bd4b14d39b87f105702c613e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17ee9f2434c74a02ad9a59074002db04 in term 0.
I20240502 14:06:29.713944 15574 leader_election.cc:302] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17ee9f2434c74a02ad9a59074002db04, ab642c5bd4b14d39b87f105702c613e5; no voters:
I20240502 14:06:29.714335 15803 tablet_bootstrap.cc:492] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04: No bootstrap required, opened a new log
I20240502 14:06:29.714745 15810 raft_consensus.cc:2781] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:29.714828 15803 ts_tablet_manager.cc:1367] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.007s
I20240502 14:06:29.715221 15810 raft_consensus.cc:482] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:29.715611 15810 raft_consensus.cc:3037] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:29.717743 15803 raft_consensus.cc:348] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:29.718515 15803 raft_consensus.cc:374] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:29.718647 15804 tablet_bootstrap.cc:492] T 56ac0dbb1eef484bba6413e33b6729b3 P f047b33669e648578b35878837bb0494: No bootstrap required, opened a new log
I20240502 14:06:29.718884 15803 raft_consensus.cc:729] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17ee9f2434c74a02ad9a59074002db04, State: Initialized, Role: FOLLOWER
I20240502 14:06:29.719391 15804 ts_tablet_manager.cc:1367] T 56ac0dbb1eef484bba6413e33b6729b3 P f047b33669e648578b35878837bb0494: Time spent bootstrapping tablet: real 0.079s user 0.049s sys 0.019s
I20240502 14:06:29.719709 15803 consensus_queue.cc:260] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:29.721740 15803 ts_tablet_manager.cc:1398] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 14:06:29.722059 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a727b7623ba4587bb183d2ce121ceed" candidate_uuid: "17ee9f2434c74a02ad9a59074002db04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f047b33669e648578b35878837bb0494" is_pre_election: true
I20240502 14:06:29.722223 15804 raft_consensus.cc:348] T 56ac0dbb1eef484bba6413e33b6729b3 P f047b33669e648578b35878837bb0494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:29.722769 15758 raft_consensus.cc:2454] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17ee9f2434c74a02ad9a59074002db04 in term 0.
I20240502 14:06:29.723146 15804 raft_consensus.cc:374] T 56ac0dbb1eef484bba6413e33b6729b3 P f047b33669e648578b35878837bb0494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:29.723500 15804 raft_consensus.cc:729] T 56ac0dbb1eef484bba6413e33b6729b3 P f047b33669e648578b35878837bb0494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f047b33669e648578b35878837bb0494, State: Initialized, Role: FOLLOWER
I20240502 14:06:29.723788 15805 tablet_bootstrap.cc:654] T dd54f15c1b87413c8541c924088c1cde P ab642c5bd4b14d39b87f105702c613e5: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:29.724411 15804 consensus_queue.cc:260] T 56ac0dbb1eef484bba6413e33b6729b3 P f047b33669e648578b35878837bb0494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:29.725142 15813 raft_consensus.cc:482] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:29.725636 15813 raft_consensus.cc:504] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:29.727733 15813 leader_election.cc:288] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab642c5bd4b14d39b87f105702c613e5 (127.11.9.66:33825), f047b33669e648578b35878837bb0494 (127.11.9.67:38309)
I20240502 14:06:29.728595 15804 ts_tablet_manager.cc:1398] T 56ac0dbb1eef484bba6413e33b6729b3 P f047b33669e648578b35878837bb0494: Time spent starting tablet: real 0.009s user 0.003s sys 0.003s
I20240502 14:06:29.729103 15684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ac0dbb1eef484bba6413e33b6729b3" candidate_uuid: "17ee9f2434c74a02ad9a59074002db04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab642c5bd4b14d39b87f105702c613e5" is_pre_election: true
I20240502 14:06:29.729223 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ac0dbb1eef484bba6413e33b6729b3" candidate_uuid: "17ee9f2434c74a02ad9a59074002db04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f047b33669e648578b35878837bb0494" is_pre_election: true
I20240502 14:06:29.729596 15804 tablet_bootstrap.cc:492] T c808defc7a8b4ecda4490809db5f983b P f047b33669e648578b35878837bb0494: Bootstrap starting.
I20240502 14:06:29.729667 15684 raft_consensus.cc:2454] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17ee9f2434c74a02ad9a59074002db04 in term 0.
I20240502 14:06:29.729872 15758 raft_consensus.cc:2454] T 56ac0dbb1eef484bba6413e33b6729b3 P f047b33669e648578b35878837bb0494 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17ee9f2434c74a02ad9a59074002db04 in term 0.
I20240502 14:06:29.731215 15572 leader_election.cc:302] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17ee9f2434c74a02ad9a59074002db04, f047b33669e648578b35878837bb0494; no voters:
I20240502 14:06:29.732156 15813 raft_consensus.cc:2781] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:29.732460 15813 raft_consensus.cc:482] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:29.732703 15813 raft_consensus.cc:3037] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:29.735862 15804 tablet_bootstrap.cc:654] T c808defc7a8b4ecda4490809db5f983b P f047b33669e648578b35878837bb0494: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:29.737829 15813 raft_consensus.cc:504] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:29.739698 15813 leader_election.cc:288] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04 [CANDIDATE]: Term 1 election: Requested vote from peers ab642c5bd4b14d39b87f105702c613e5 (127.11.9.66:33825), f047b33669e648578b35878837bb0494 (127.11.9.67:38309)
I20240502 14:06:29.740322 15684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ac0dbb1eef484bba6413e33b6729b3" candidate_uuid: "17ee9f2434c74a02ad9a59074002db04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab642c5bd4b14d39b87f105702c613e5"
I20240502 14:06:29.740790 15684 raft_consensus.cc:3037] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:29.740844 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ac0dbb1eef484bba6413e33b6729b3" candidate_uuid: "17ee9f2434c74a02ad9a59074002db04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f047b33669e648578b35878837bb0494"
I20240502 14:06:29.741410 15758 raft_consensus.cc:3037] T 56ac0dbb1eef484bba6413e33b6729b3 P f047b33669e648578b35878837bb0494 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:29.737756 15810 raft_consensus.cc:504] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:29.747668 15684 raft_consensus.cc:2454] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17ee9f2434c74a02ad9a59074002db04 in term 1.
I20240502 14:06:29.747548 15683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a727b7623ba4587bb183d2ce121ceed" candidate_uuid: "17ee9f2434c74a02ad9a59074002db04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab642c5bd4b14d39b87f105702c613e5"
I20240502 14:06:29.748477 15683 raft_consensus.cc:3037] T 8a727b7623ba4587bb183d2ce121ceed P ab642c5bd4b14d39b87f105702c613e5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:29.748762 15758 raft_consensus.cc:2454] T 56ac0dbb1eef484bba6413e33b6729b3 P f047b33669e648578b35878837bb0494 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17ee9f2434c74a02ad9a59074002db04 in term 1.
I20240502 14:06:29.748817 15574 leader_election.cc:302] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17ee9f2434c74a02ad9a59074002db04, ab642c5bd4b14d39b87f105702c613e5; no voters:
I20240502 14:06:29.749922 15813 raft_consensus.cc:2781] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:29.754637 15813 raft_consensus.cc:686] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04 [term 1 LEADER]: Becoming Leader. State: Replica: 17ee9f2434c74a02ad9a59074002db04, State: Running, Role: LEADER
I20240502 14:06:29.755437 15813 consensus_queue.cc:237] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:29.761749 15810 leader_election.cc:288] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04 [CANDIDATE]: Term 1 election: Requested vote from peers ab642c5bd4b14d39b87f105702c613e5 (127.11.9.66:33825), f047b33669e648578b35878837bb0494 (127.11.9.67:38309)
I20240502 14:06:29.763054 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a727b7623ba4587bb183d2ce121ceed" candidate_uuid: "17ee9f2434c74a02ad9a59074002db04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f047b33669e648578b35878837bb0494"
I20240502 14:06:29.763830 15758 raft_consensus.cc:3037] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:29.764277 15805 tablet_bootstrap.cc:492] T dd54f15c1b87413c8541c924088c1cde P ab642c5bd4b14d39b87f105702c613e5: No bootstrap required, opened a new log
I20240502 14:06:29.764727 15805 ts_tablet_manager.cc:1367] T dd54f15c1b87413c8541c924088c1cde P ab642c5bd4b14d39b87f105702c613e5: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.008s
I20240502 14:06:29.764046 15506 catalog_manager.cc:5484] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65). New cstate: current_term: 1 leader_uuid: "17ee9f2434c74a02ad9a59074002db04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:29.767343 15805 raft_consensus.cc:348] T dd54f15c1b87413c8541c924088c1cde P ab642c5bd4b14d39b87f105702c613e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:29.767900 15805 raft_consensus.cc:374] T dd54f15c1b87413c8541c924088c1cde P ab642c5bd4b14d39b87f105702c613e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:29.768110 15805 raft_consensus.cc:729] T dd54f15c1b87413c8541c924088c1cde P ab642c5bd4b14d39b87f105702c613e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab642c5bd4b14d39b87f105702c613e5, State: Initialized, Role: FOLLOWER
I20240502 14:06:29.768671 15805 consensus_queue.cc:260] T dd54f15c1b87413c8541c924088c1cde P ab642c5bd4b14d39b87f105702c613e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:29.770551 15805 ts_tablet_manager.cc:1398] T dd54f15c1b87413c8541c924088c1cde P ab642c5bd4b14d39b87f105702c613e5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240502 14:06:29.773695 15804 tablet_bootstrap.cc:492] T c808defc7a8b4ecda4490809db5f983b P f047b33669e648578b35878837bb0494: No bootstrap required, opened a new log
I20240502 14:06:29.774169 15804 ts_tablet_manager.cc:1367] T c808defc7a8b4ecda4490809db5f983b P f047b33669e648578b35878837bb0494: Time spent bootstrapping tablet: real 0.045s user 0.032s sys 0.011s
I20240502 14:06:29.761507 15683 raft_consensus.cc:2454] T 8a727b7623ba4587bb183d2ce121ceed P ab642c5bd4b14d39b87f105702c613e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17ee9f2434c74a02ad9a59074002db04 in term 1.
I20240502 14:06:29.775409 15574 leader_election.cc:302] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17ee9f2434c74a02ad9a59074002db04, ab642c5bd4b14d39b87f105702c613e5; no voters:
I20240502 14:06:29.775812 15758 raft_consensus.cc:2454] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17ee9f2434c74a02ad9a59074002db04 in term 1.
I20240502 14:06:29.776260 15810 raft_consensus.cc:2781] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:29.776728 15810 raft_consensus.cc:686] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04 [term 1 LEADER]: Becoming Leader. State: Replica: 17ee9f2434c74a02ad9a59074002db04, State: Running, Role: LEADER
I20240502 14:06:29.777063 15812 raft_consensus.cc:482] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:29.777066 15804 raft_consensus.cc:348] T c808defc7a8b4ecda4490809db5f983b P f047b33669e648578b35878837bb0494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } }
I20240502 14:06:29.777473 15810 consensus_queue.cc:237] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:29.777702 15812 raft_consensus.cc:504] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } }
I20240502 14:06:29.778090 15804 raft_consensus.cc:374] T c808defc7a8b4ecda4490809db5f983b P f047b33669e648578b35878837bb0494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:29.778548 15804 raft_consensus.cc:729] T c808defc7a8b4ecda4490809db5f983b P f047b33669e648578b35878837bb0494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f047b33669e648578b35878837bb0494, State: Initialized, Role: FOLLOWER
I20240502 14:06:29.780289 15812 leader_election.cc:288] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005), f047b33669e648578b35878837bb0494 (127.11.9.67:38309)
I20240502 14:06:29.779323 15804 consensus_queue.cc:260] T c808defc7a8b4ecda4490809db5f983b P f047b33669e648578b35878837bb0494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } }
I20240502 14:06:29.787796 15804 ts_tablet_manager.cc:1398] T c808defc7a8b4ecda4490809db5f983b P f047b33669e648578b35878837bb0494: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20240502 14:06:29.797048 15507 catalog_manager.cc:5484] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65). New cstate: current_term: 1 leader_uuid: "17ee9f2434c74a02ad9a59074002db04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:29.803452 15609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c808defc7a8b4ecda4490809db5f983b" candidate_uuid: "ab642c5bd4b14d39b87f105702c613e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17ee9f2434c74a02ad9a59074002db04" is_pre_election: true
I20240502 14:06:29.804129 15609 raft_consensus.cc:2454] T c808defc7a8b4ecda4490809db5f983b P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab642c5bd4b14d39b87f105702c613e5 in term 0.
I20240502 14:06:29.805378 15649 leader_election.cc:302] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17ee9f2434c74a02ad9a59074002db04, ab642c5bd4b14d39b87f105702c613e5; no voters:
I20240502 14:06:29.805588 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c808defc7a8b4ecda4490809db5f983b" candidate_uuid: "ab642c5bd4b14d39b87f105702c613e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f047b33669e648578b35878837bb0494" is_pre_election: true
I20240502 14:06:29.806107 15812 raft_consensus.cc:2781] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:29.806295 15758 raft_consensus.cc:2454] T c808defc7a8b4ecda4490809db5f983b P f047b33669e648578b35878837bb0494 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab642c5bd4b14d39b87f105702c613e5 in term 0.
I20240502 14:06:29.806447 15812 raft_consensus.cc:482] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:29.806833 15812 raft_consensus.cc:3037] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:29.811342 15812 raft_consensus.cc:504] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } }
I20240502 14:06:29.812760 15812 leader_election.cc:288] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [CANDIDATE]: Term 1 election: Requested vote from peers 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005), f047b33669e648578b35878837bb0494 (127.11.9.67:38309)
I20240502 14:06:29.813391 15609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c808defc7a8b4ecda4490809db5f983b" candidate_uuid: "ab642c5bd4b14d39b87f105702c613e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17ee9f2434c74a02ad9a59074002db04"
I20240502 14:06:29.813654 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c808defc7a8b4ecda4490809db5f983b" candidate_uuid: "ab642c5bd4b14d39b87f105702c613e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f047b33669e648578b35878837bb0494"
I20240502 14:06:29.813889 15609 raft_consensus.cc:3037] T c808defc7a8b4ecda4490809db5f983b P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:29.814149 15758 raft_consensus.cc:3037] T c808defc7a8b4ecda4490809db5f983b P f047b33669e648578b35878837bb0494 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:29.818368 15609 raft_consensus.cc:2454] T c808defc7a8b4ecda4490809db5f983b P 17ee9f2434c74a02ad9a59074002db04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab642c5bd4b14d39b87f105702c613e5 in term 1.
I20240502 14:06:29.818480 15758 raft_consensus.cc:2454] T c808defc7a8b4ecda4490809db5f983b P f047b33669e648578b35878837bb0494 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab642c5bd4b14d39b87f105702c613e5 in term 1.
I20240502 14:06:29.819172 15649 leader_election.cc:302] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17ee9f2434c74a02ad9a59074002db04, ab642c5bd4b14d39b87f105702c613e5; no voters:
I20240502 14:06:29.819773 15812 raft_consensus.cc:2781] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:29.820771 15812 raft_consensus.cc:686] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [term 1 LEADER]: Becoming Leader. State: Replica: ab642c5bd4b14d39b87f105702c613e5, State: Running, Role: LEADER
I20240502 14:06:29.821434 15812 consensus_queue.cc:237] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } }
I20240502 14:06:29.827374 15507 catalog_manager.cc:5484] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab642c5bd4b14d39b87f105702c613e5 (127.11.9.66). New cstate: current_term: 1 leader_uuid: "ab642c5bd4b14d39b87f105702c613e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } health_report { overall_health: HEALTHY } } }
W20240502 14:06:30.047080 15555 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 14:06:30.050076 15810 raft_consensus.cc:482] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:30.050483 15810 raft_consensus.cc:504] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:30.052091 15810 leader_election.cc:288] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab642c5bd4b14d39b87f105702c613e5 (127.11.9.66:33825), f047b33669e648578b35878837bb0494 (127.11.9.67:38309)
I20240502 14:06:30.052863 15683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd54f15c1b87413c8541c924088c1cde" candidate_uuid: "17ee9f2434c74a02ad9a59074002db04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab642c5bd4b14d39b87f105702c613e5" is_pre_election: true
I20240502 14:06:30.053072 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd54f15c1b87413c8541c924088c1cde" candidate_uuid: "17ee9f2434c74a02ad9a59074002db04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f047b33669e648578b35878837bb0494" is_pre_election: true
I20240502 14:06:30.053414 15683 raft_consensus.cc:2454] T dd54f15c1b87413c8541c924088c1cde P ab642c5bd4b14d39b87f105702c613e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17ee9f2434c74a02ad9a59074002db04 in term 0.
I20240502 14:06:30.053659 15758 raft_consensus.cc:2454] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17ee9f2434c74a02ad9a59074002db04 in term 0.
I20240502 14:06:30.054334 15574 leader_election.cc:302] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17ee9f2434c74a02ad9a59074002db04, ab642c5bd4b14d39b87f105702c613e5; no voters:
I20240502 14:06:30.054988 15810 raft_consensus.cc:2781] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:30.055342 15810 raft_consensus.cc:482] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:30.055598 15810 raft_consensus.cc:3037] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:30.059940 15810 raft_consensus.cc:504] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:30.061347 15810 leader_election.cc:288] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04 [CANDIDATE]: Term 1 election: Requested vote from peers ab642c5bd4b14d39b87f105702c613e5 (127.11.9.66:33825), f047b33669e648578b35878837bb0494 (127.11.9.67:38309)
I20240502 14:06:30.062027 15683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd54f15c1b87413c8541c924088c1cde" candidate_uuid: "17ee9f2434c74a02ad9a59074002db04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab642c5bd4b14d39b87f105702c613e5"
I20240502 14:06:30.062258 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd54f15c1b87413c8541c924088c1cde" candidate_uuid: "17ee9f2434c74a02ad9a59074002db04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f047b33669e648578b35878837bb0494"
I20240502 14:06:30.062501 15683 raft_consensus.cc:3037] T dd54f15c1b87413c8541c924088c1cde P ab642c5bd4b14d39b87f105702c613e5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:30.062786 15758 raft_consensus.cc:3037] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:30.066668 15683 raft_consensus.cc:2454] T dd54f15c1b87413c8541c924088c1cde P ab642c5bd4b14d39b87f105702c613e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17ee9f2434c74a02ad9a59074002db04 in term 1.
I20240502 14:06:30.067086 15758 raft_consensus.cc:2454] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17ee9f2434c74a02ad9a59074002db04 in term 1.
I20240502 14:06:30.067544 15574 leader_election.cc:302] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17ee9f2434c74a02ad9a59074002db04, ab642c5bd4b14d39b87f105702c613e5; no voters:
I20240502 14:06:30.068259 15810 raft_consensus.cc:2781] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:30.068646 15810 raft_consensus.cc:686] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04 [term 1 LEADER]: Becoming Leader. State: Replica: 17ee9f2434c74a02ad9a59074002db04, State: Running, Role: LEADER
I20240502 14:06:30.069167 15810 consensus_queue.cc:237] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:30.075850 15507 catalog_manager.cc:5484] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65). New cstate: current_term: 1 leader_uuid: "17ee9f2434c74a02ad9a59074002db04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:30.215251 11301 tablet_server.cc:178] TabletServer@127.11.9.65:0 shutting down...
I20240502 14:06:30.231580 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:06:30.232139 11301 tablet_replica.cc:330] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04: stopping tablet replica
I20240502 14:06:30.232626 11301 raft_consensus.cc:2229] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:30.233147 11301 pending_rounds.cc:62] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04: Trying to abort 1 pending ops.
I20240502 14:06:30.233340 11301 pending_rounds.cc:68] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023242404129415168 op_type: NO_OP noop_request { }
I20240502 14:06:30.233647 11301 raft_consensus.cc:2866] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 14:06:30.233870 11301 raft_consensus.cc:2258] T dd54f15c1b87413c8541c924088c1cde P 17ee9f2434c74a02ad9a59074002db04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:30.235675 11301 tablet_replica.cc:330] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04: stopping tablet replica
I20240502 14:06:30.236120 11301 raft_consensus.cc:2229] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:30.236655 11301 pending_rounds.cc:62] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04: Trying to abort 1 pending ops.
I20240502 14:06:30.236810 11301 pending_rounds.cc:68] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023242402845548544 op_type: NO_OP noop_request { }
I20240502 14:06:30.237030 11301 raft_consensus.cc:2866] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 14:06:30.237246 11301 raft_consensus.cc:2258] T 56ac0dbb1eef484bba6413e33b6729b3 P 17ee9f2434c74a02ad9a59074002db04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:30.239023 11301 tablet_replica.cc:330] T c808defc7a8b4ecda4490809db5f983b P 17ee9f2434c74a02ad9a59074002db04: stopping tablet replica
I20240502 14:06:30.239416 11301 raft_consensus.cc:2229] T c808defc7a8b4ecda4490809db5f983b P 17ee9f2434c74a02ad9a59074002db04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:30.239670 11301 raft_consensus.cc:2258] T c808defc7a8b4ecda4490809db5f983b P 17ee9f2434c74a02ad9a59074002db04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:30.241004 11301 tablet_replica.cc:330] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04: stopping tablet replica
I20240502 14:06:30.241385 11301 raft_consensus.cc:2229] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:30.241852 11301 pending_rounds.cc:62] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04: Trying to abort 1 pending ops.
I20240502 14:06:30.241997 11301 pending_rounds.cc:68] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023242402937946112 op_type: NO_OP noop_request { }
I20240502 14:06:30.242208 11301 raft_consensus.cc:2866] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 14:06:30.242390 11301 raft_consensus.cc:2258] T 8a727b7623ba4587bb183d2ce121ceed P 17ee9f2434c74a02ad9a59074002db04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:30.253237 11301 tablet_server.cc:194] TabletServer@127.11.9.65:0 shutdown complete.
I20240502 14:06:30.264303 15812 consensus_queue.cc:1035] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:06:30.274060 15649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:06:30.278045 15649 consensus_peers.cc:485] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 -> Peer 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005): Couldn't send request to peer 17ee9f2434c74a02ad9a59074002db04. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:30.884112 15812 consensus_queue.cc:579] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [LEADER]: Leader has been unable to successfully communicate with peer 17ee9f2434c74a02ad9a59074002db04 for more than 1 seconds (1.062s)
W20240502 14:06:30.916448 15506 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c808defc7a8b4ecda4490809db5f983b with cas_config_opid_index -1: no extra replica candidate found for tablet c808defc7a8b4ecda4490809db5f983b (table test-workload [id=b7e312dadc20472d9c58741722e9b268]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20240502 14:06:31.264652 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:31.271726 15841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:31.273979 15842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:31.274206 15844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:31.274940 11301 server_base.cc:1034] running on GCE node
I20240502 14:06:31.276576 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:06:31.276818 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:31.277001 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658791276983 us; error 0 us; skew 500 ppm
I20240502 14:06:31.277585 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:31.292402 11301 webserver.cc:460] Webserver started at http://127.11.9.68:40637/ using document root <none> and password file <none>
I20240502 14:06:31.292922 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:31.293129 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:31.293421 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:31.294929 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714658659912502-11301-0/minicluster-data/ts-3-root/instance:
uuid: "89449202af1248fb826e51c95e85d882"
format_stamp: "Formatted at 2024-05-02 14:06:31 on dist-test-slave-j72s"
I20240502 14:06:31.299851 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 14:06:31.303256 15849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:31.304049 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:06:31.304283 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714658659912502-11301-0/minicluster-data/ts-3-root
uuid: "89449202af1248fb826e51c95e85d882"
format_stamp: "Formatted at 2024-05-02 14:06:31 on dist-test-slave-j72s"
I20240502 14:06:31.304518 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714658659912502-11301-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714658659912502-11301-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714658659912502-11301-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:31.326352 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:31.327459 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:31.328787 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:31.331244 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:31.331442 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:31.331660 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:31.331846 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:31.370409 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.68:45769
I20240502 14:06:31.370527 15911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.68:45769 every 8 connection(s)
I20240502 14:06:31.384735 15912 heartbeater.cc:347] Connected to a master server at 127.11.9.126:40531
I20240502 14:06:31.385073 15912 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:31.385794 15912 heartbeater.cc:511] Master 127.11.9.126:40531 requested a full tablet report, sending...
I20240502 14:06:31.387697 15506 ts_manager.cc:194] Registered new tserver with Master: 89449202af1248fb826e51c95e85d882 (127.11.9.68:45769)
I20240502 14:06:31.389127 15506 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34974
I20240502 14:06:31.562306 15683 consensus_queue.cc:237] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: true } }
I20240502 14:06:31.568939 15758 raft_consensus.cc:1261] T c808defc7a8b4ecda4490809db5f983b P f047b33669e648578b35878837bb0494 [term 1 FOLLOWER]: Refusing update from remote peer ab642c5bd4b14d39b87f105702c613e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:06:31.571197 15838 consensus_queue.cc:1035] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:06:31.575435 15649 consensus_peers.cc:485] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 -> Peer 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005): Couldn't send request to peer 17ee9f2434c74a02ad9a59074002db04. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:31.579826 15919 raft_consensus.cc:2932] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 89449202af1248fb826e51c95e85d882 (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: true } } }
I20240502 14:06:31.581329 15758 raft_consensus.cc:2932] T c808defc7a8b4ecda4490809db5f983b P f047b33669e648578b35878837bb0494 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 89449202af1248fb826e51c95e85d882 (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: true } } }
W20240502 14:06:31.582253 15649 consensus_peers.cc:485] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 -> Peer 89449202af1248fb826e51c95e85d882 (127.11.9.68:45769): Couldn't send request to peer 89449202af1248fb826e51c95e85d882. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c808defc7a8b4ecda4490809db5f983b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:31.589340 15493 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c808defc7a8b4ecda4490809db5f983b with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
I20240502 14:06:31.597268 15506 catalog_manager.cc:5484] T c808defc7a8b4ecda4490809db5f983b P f047b33669e648578b35878837bb0494 reported cstate change: config changed from index -1 to 2, NON_VOTER 89449202af1248fb826e51c95e85d882 (127.11.9.68) added. New cstate: current_term: 1 leader_uuid: "ab642c5bd4b14d39b87f105702c613e5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: true } } }
I20240502 14:06:31.649901 15923 raft_consensus.cc:482] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:31.650368 15923 raft_consensus.cc:504] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:31.652164 15923 leader_election.cc:288] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005), ab642c5bd4b14d39b87f105702c613e5 (127.11.9.66:33825)
W20240502 14:06:31.658802 15723 leader_election.cc:334] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005): Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111)
I20240502 14:06:31.663552 15683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a727b7623ba4587bb183d2ce121ceed" candidate_uuid: "f047b33669e648578b35878837bb0494" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab642c5bd4b14d39b87f105702c613e5" is_pre_election: true
I20240502 14:06:31.664167 15683 raft_consensus.cc:2454] T 8a727b7623ba4587bb183d2ce121ceed P ab642c5bd4b14d39b87f105702c613e5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f047b33669e648578b35878837bb0494 in term 1.
I20240502 14:06:31.664996 15723 leader_election.cc:302] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab642c5bd4b14d39b87f105702c613e5, f047b33669e648578b35878837bb0494; no voters: 17ee9f2434c74a02ad9a59074002db04
I20240502 14:06:31.665623 15923 raft_consensus.cc:2781] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:06:31.665890 15923 raft_consensus.cc:482] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:31.666123 15923 raft_consensus.cc:3037] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:31.671236 15923 raft_consensus.cc:504] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:31.672610 15923 leader_election.cc:288] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [CANDIDATE]: Term 2 election: Requested vote from peers 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005), ab642c5bd4b14d39b87f105702c613e5 (127.11.9.66:33825)
I20240502 14:06:31.674005 15683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a727b7623ba4587bb183d2ce121ceed" candidate_uuid: "f047b33669e648578b35878837bb0494" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab642c5bd4b14d39b87f105702c613e5"
I20240502 14:06:31.674551 15683 raft_consensus.cc:3037] T 8a727b7623ba4587bb183d2ce121ceed P ab642c5bd4b14d39b87f105702c613e5 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:06:31.675562 15723 leader_election.cc:334] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005): Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111)
I20240502 14:06:31.679533 15683 raft_consensus.cc:2454] T 8a727b7623ba4587bb183d2ce121ceed P ab642c5bd4b14d39b87f105702c613e5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f047b33669e648578b35878837bb0494 in term 2.
I20240502 14:06:31.680323 15723 leader_election.cc:302] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab642c5bd4b14d39b87f105702c613e5, f047b33669e648578b35878837bb0494; no voters: 17ee9f2434c74a02ad9a59074002db04
I20240502 14:06:31.680842 15923 raft_consensus.cc:2781] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:06:31.681771 15923 raft_consensus.cc:686] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [term 2 LEADER]: Becoming Leader. State: Replica: f047b33669e648578b35878837bb0494, State: Running, Role: LEADER
I20240502 14:06:31.682602 15923 consensus_queue.cc:237] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:31.688630 15507 catalog_manager.cc:5484] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 reported cstate change: term changed from 1 to 2, leader changed from 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65) to f047b33669e648578b35878837bb0494 (127.11.9.67). New cstate: current_term: 2 leader_uuid: "f047b33669e648578b35878837bb0494" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } health_report { overall_health: HEALTHY } } }
I20240502 14:06:31.694829 15838 raft_consensus.cc:482] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:31.695233 15838 raft_consensus.cc:504] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:31.696875 15838 leader_election.cc:288] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005), f047b33669e648578b35878837bb0494 (127.11.9.67:38309)
I20240502 14:06:31.697811 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ac0dbb1eef484bba6413e33b6729b3" candidate_uuid: "ab642c5bd4b14d39b87f105702c613e5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f047b33669e648578b35878837bb0494" is_pre_election: true
I20240502 14:06:31.698344 15758 raft_consensus.cc:2454] T 56ac0dbb1eef484bba6413e33b6729b3 P f047b33669e648578b35878837bb0494 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab642c5bd4b14d39b87f105702c613e5 in term 1.
I20240502 14:06:31.699308 15647 leader_election.cc:302] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab642c5bd4b14d39b87f105702c613e5, f047b33669e648578b35878837bb0494; no voters:
W20240502 14:06:31.699735 15649 leader_election.cc:334] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005): Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111)
I20240502 14:06:31.699877 15838 raft_consensus.cc:2781] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:06:31.700172 15838 raft_consensus.cc:482] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:31.700413 15838 raft_consensus.cc:3037] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:31.705022 15838 raft_consensus.cc:504] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:31.706593 15838 leader_election.cc:288] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [CANDIDATE]: Term 2 election: Requested vote from peers 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005), f047b33669e648578b35878837bb0494 (127.11.9.67:38309)
I20240502 14:06:31.707474 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ac0dbb1eef484bba6413e33b6729b3" candidate_uuid: "ab642c5bd4b14d39b87f105702c613e5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f047b33669e648578b35878837bb0494"
I20240502 14:06:31.707967 15758 raft_consensus.cc:3037] T 56ac0dbb1eef484bba6413e33b6729b3 P f047b33669e648578b35878837bb0494 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:06:31.709198 15649 leader_election.cc:334] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005): Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111)
I20240502 14:06:31.712136 15758 raft_consensus.cc:2454] T 56ac0dbb1eef484bba6413e33b6729b3 P f047b33669e648578b35878837bb0494 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab642c5bd4b14d39b87f105702c613e5 in term 2.
I20240502 14:06:31.712947 15647 leader_election.cc:302] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab642c5bd4b14d39b87f105702c613e5, f047b33669e648578b35878837bb0494; no voters: 17ee9f2434c74a02ad9a59074002db04
I20240502 14:06:31.713546 15838 raft_consensus.cc:2781] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:06:31.713905 15838 raft_consensus.cc:686] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [term 2 LEADER]: Becoming Leader. State: Replica: ab642c5bd4b14d39b87f105702c613e5, State: Running, Role: LEADER
I20240502 14:06:31.714645 15838 consensus_queue.cc:237] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:31.721406 15507 catalog_manager.cc:5484] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 reported cstate change: term changed from 1 to 2, leader changed from 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65) to ab642c5bd4b14d39b87f105702c613e5 (127.11.9.66). New cstate: current_term: 2 leader_uuid: "ab642c5bd4b14d39b87f105702c613e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:31.981186 15937 ts_tablet_manager.cc:902] T c808defc7a8b4ecda4490809db5f983b P 89449202af1248fb826e51c95e85d882: Initiating tablet copy from peer ab642c5bd4b14d39b87f105702c613e5 (127.11.9.66:33825)
I20240502 14:06:31.984031 15937 tablet_copy_client.cc:308] T c808defc7a8b4ecda4490809db5f983b P 89449202af1248fb826e51c95e85d882: tablet copy: Beginning tablet copy session from remote peer at address 127.11.9.66:33825
I20240502 14:06:31.992669 15694 tablet_copy_service.cc:140] P ab642c5bd4b14d39b87f105702c613e5: Received BeginTabletCopySession request for tablet c808defc7a8b4ecda4490809db5f983b from peer 89449202af1248fb826e51c95e85d882 ({username='slave'} at 127.0.0.1:46764)
I20240502 14:06:31.993224 15694 tablet_copy_service.cc:161] P ab642c5bd4b14d39b87f105702c613e5: Beginning new tablet copy session on tablet c808defc7a8b4ecda4490809db5f983b from peer 89449202af1248fb826e51c95e85d882 at {username='slave'} at 127.0.0.1:46764: session id = 89449202af1248fb826e51c95e85d882-c808defc7a8b4ecda4490809db5f983b
I20240502 14:06:31.999589 15694 tablet_copy_source_session.cc:205] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:06:32.002131 15937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c808defc7a8b4ecda4490809db5f983b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:32.012354 15937 tablet_copy_client.cc:783] T c808defc7a8b4ecda4490809db5f983b P 89449202af1248fb826e51c95e85d882: tablet copy: Starting download of 0 data blocks...
I20240502 14:06:32.012782 15937 tablet_copy_client.cc:647] T c808defc7a8b4ecda4490809db5f983b P 89449202af1248fb826e51c95e85d882: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:32.015784 15937 tablet_copy_client.cc:523] T c808defc7a8b4ecda4490809db5f983b P 89449202af1248fb826e51c95e85d882: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:32.021199 15937 tablet_bootstrap.cc:492] T c808defc7a8b4ecda4490809db5f983b P 89449202af1248fb826e51c95e85d882: Bootstrap starting.
I20240502 14:06:32.034792 15937 tablet_bootstrap.cc:492] T c808defc7a8b4ecda4490809db5f983b P 89449202af1248fb826e51c95e85d882: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:32.035414 15937 tablet_bootstrap.cc:492] T c808defc7a8b4ecda4490809db5f983b P 89449202af1248fb826e51c95e85d882: Bootstrap complete.
I20240502 14:06:32.035925 15937 ts_tablet_manager.cc:1367] T c808defc7a8b4ecda4490809db5f983b P 89449202af1248fb826e51c95e85d882: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.008s
I20240502 14:06:32.037902 15937 raft_consensus.cc:348] T c808defc7a8b4ecda4490809db5f983b P 89449202af1248fb826e51c95e85d882 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: true } }
I20240502 14:06:32.038461 15937 raft_consensus.cc:729] T c808defc7a8b4ecda4490809db5f983b P 89449202af1248fb826e51c95e85d882 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 89449202af1248fb826e51c95e85d882, State: Initialized, Role: LEARNER
I20240502 14:06:32.038951 15937 consensus_queue.cc:260] T c808defc7a8b4ecda4490809db5f983b P 89449202af1248fb826e51c95e85d882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: true } }
I20240502 14:06:32.040797 15912 heartbeater.cc:503] Master 127.11.9.126:40531 was elected leader, sending a full tablet report...
I20240502 14:06:32.041049 15937 ts_tablet_manager.cc:1398] T c808defc7a8b4ecda4490809db5f983b P 89449202af1248fb826e51c95e85d882: Time spent starting tablet: real 0.005s user 0.002s sys 0.005s
I20240502 14:06:32.042763 15694 tablet_copy_service.cc:342] P ab642c5bd4b14d39b87f105702c613e5: Request end of tablet copy session 89449202af1248fb826e51c95e85d882-c808defc7a8b4ecda4490809db5f983b received from {username='slave'} at 127.0.0.1:46764
I20240502 14:06:32.043148 15694 tablet_copy_service.cc:433] P ab642c5bd4b14d39b87f105702c613e5: ending tablet copy session 89449202af1248fb826e51c95e85d882-c808defc7a8b4ecda4490809db5f983b on tablet c808defc7a8b4ecda4490809db5f983b with peer 89449202af1248fb826e51c95e85d882
W20240502 14:06:32.055711 15555 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 17ee9f2434c74a02ad9a59074002db04 not available for placement
W20240502 14:06:32.103489 15723 consensus_peers.cc:485] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 -> Peer 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005): Couldn't send request to peer 17ee9f2434c74a02ad9a59074002db04. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:32.139544 15923 raft_consensus.cc:482] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:32.140038 15923 raft_consensus.cc:504] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:32.141635 15923 leader_election.cc:288] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005), ab642c5bd4b14d39b87f105702c613e5 (127.11.9.66:33825)
I20240502 14:06:32.143033 15683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd54f15c1b87413c8541c924088c1cde" candidate_uuid: "f047b33669e648578b35878837bb0494" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab642c5bd4b14d39b87f105702c613e5" is_pre_election: true
I20240502 14:06:32.143679 15683 raft_consensus.cc:2454] T dd54f15c1b87413c8541c924088c1cde P ab642c5bd4b14d39b87f105702c613e5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f047b33669e648578b35878837bb0494 in term 1.
I20240502 14:06:32.144487 15723 leader_election.cc:302] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab642c5bd4b14d39b87f105702c613e5, f047b33669e648578b35878837bb0494; no voters:
I20240502 14:06:32.145021 15923 raft_consensus.cc:2781] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240502 14:06:32.145059 15723 leader_election.cc:334] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005): Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111)
I20240502 14:06:32.145360 15923 raft_consensus.cc:482] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:32.145741 15923 raft_consensus.cc:3037] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:32.150038 15923 raft_consensus.cc:504] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:32.151376 15923 leader_election.cc:288] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [CANDIDATE]: Term 2 election: Requested vote from peers 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005), ab642c5bd4b14d39b87f105702c613e5 (127.11.9.66:33825)
I20240502 14:06:32.152673 15683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd54f15c1b87413c8541c924088c1cde" candidate_uuid: "f047b33669e648578b35878837bb0494" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab642c5bd4b14d39b87f105702c613e5"
I20240502 14:06:32.153229 15683 raft_consensus.cc:3037] T dd54f15c1b87413c8541c924088c1cde P ab642c5bd4b14d39b87f105702c613e5 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:06:32.154191 15723 leader_election.cc:334] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005): Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111)
I20240502 14:06:32.157586 15683 raft_consensus.cc:2454] T dd54f15c1b87413c8541c924088c1cde P ab642c5bd4b14d39b87f105702c613e5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f047b33669e648578b35878837bb0494 in term 2.
I20240502 14:06:32.158313 15723 leader_election.cc:302] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab642c5bd4b14d39b87f105702c613e5, f047b33669e648578b35878837bb0494; no voters: 17ee9f2434c74a02ad9a59074002db04
I20240502 14:06:32.158862 15923 raft_consensus.cc:2781] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:06:32.159349 15923 raft_consensus.cc:686] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [term 2 LEADER]: Becoming Leader. State: Replica: f047b33669e648578b35878837bb0494, State: Running, Role: LEADER
I20240502 14:06:32.160156 15923 consensus_queue.cc:237] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } }
I20240502 14:06:32.165956 15507 catalog_manager.cc:5484] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 reported cstate change: term changed from 1 to 2, leader changed from 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65) to f047b33669e648578b35878837bb0494 (127.11.9.67). New cstate: current_term: 2 leader_uuid: "f047b33669e648578b35878837bb0494" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } health_report { overall_health: HEALTHY } } }
I20240502 14:06:32.203007 15758 raft_consensus.cc:1261] T 56ac0dbb1eef484bba6413e33b6729b3 P f047b33669e648578b35878837bb0494 [term 2 FOLLOWER]: Refusing update from remote peer ab642c5bd4b14d39b87f105702c613e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 14:06:32.203935 15838 consensus_queue.cc:1035] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:06:32.215916 15649 consensus_peers.cc:485] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 -> Peer 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005): Couldn't send request to peer 17ee9f2434c74a02ad9a59074002db04. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:32.253396 15683 raft_consensus.cc:1261] T 8a727b7623ba4587bb183d2ce121ceed P ab642c5bd4b14d39b87f105702c613e5 [term 2 FOLLOWER]: Refusing update from remote peer f047b33669e648578b35878837bb0494: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 14:06:32.254511 15923 consensus_queue.cc:1035] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:32.451920 15887 raft_consensus.cc:1203] T c808defc7a8b4ecda4490809db5f983b P 89449202af1248fb826e51c95e85d882 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:06:32.635182 15683 raft_consensus.cc:1261] T dd54f15c1b87413c8541c924088c1cde P ab642c5bd4b14d39b87f105702c613e5 [term 2 FOLLOWER]: Refusing update from remote peer f047b33669e648578b35878837bb0494: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 14:06:32.636307 15923 consensus_queue.cc:1035] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:06:32.647827 15723 consensus_peers.cc:485] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 -> Peer 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005): Couldn't send request to peer 17ee9f2434c74a02ad9a59074002db04. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:32.760321 15950 consensus_queue.cc:579] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [LEADER]: Leader has been unable to successfully communicate with peer 17ee9f2434c74a02ad9a59074002db04 for more than 1 seconds (1.045s)
I20240502 14:06:32.768688 15683 consensus_queue.cc:237] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: true } }
I20240502 14:06:32.772953 15758 raft_consensus.cc:1261] T 56ac0dbb1eef484bba6413e33b6729b3 P f047b33669e648578b35878837bb0494 [term 2 FOLLOWER]: Refusing update from remote peer ab642c5bd4b14d39b87f105702c613e5: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240502 14:06:32.773991 15649 consensus_peers.cc:485] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 -> Peer 89449202af1248fb826e51c95e85d882 (127.11.9.68:45769): Couldn't send request to peer 89449202af1248fb826e51c95e85d882. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 56ac0dbb1eef484bba6413e33b6729b3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:32.774291 15924 consensus_queue.cc:1035] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:06:32.775708 15649 consensus_peers.cc:485] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 -> Peer 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005): Couldn't send request to peer 17ee9f2434c74a02ad9a59074002db04. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:32.778812 15838 raft_consensus.cc:2932] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 89449202af1248fb826e51c95e85d882 (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: true } } }
I20240502 14:06:32.780061 15758 raft_consensus.cc:2932] T 56ac0dbb1eef484bba6413e33b6729b3 P f047b33669e648578b35878837bb0494 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 89449202af1248fb826e51c95e85d882 (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: true } } }
I20240502 14:06:32.784814 15493 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 56ac0dbb1eef484bba6413e33b6729b3 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:06:32.788439 15507 catalog_manager.cc:5484] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 reported cstate change: config changed from index -1 to 2, NON_VOTER 89449202af1248fb826e51c95e85d882 (127.11.9.68) added. New cstate: current_term: 2 leader_uuid: "ab642c5bd4b14d39b87f105702c613e5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:32.814220 15838 consensus_queue.cc:579] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [LEADER]: Leader has been unable to successfully communicate with peer 17ee9f2434c74a02ad9a59074002db04 for more than 1 seconds (1.251s)
I20240502 14:06:32.990978 15923 consensus_queue.cc:579] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [LEADER]: Leader has been unable to successfully communicate with peer 17ee9f2434c74a02ad9a59074002db04 for more than 1 seconds (1.307s)
I20240502 14:06:32.999840 15758 consensus_queue.cc:237] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: true } }
I20240502 14:06:33.000389 15838 raft_consensus.cc:1050] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5: attempting to promote NON_VOTER 89449202af1248fb826e51c95e85d882 to VOTER
I20240502 14:06:33.002256 15838 consensus_queue.cc:237] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } }
I20240502 14:06:33.005203 15683 raft_consensus.cc:1261] T 8a727b7623ba4587bb183d2ce121ceed P ab642c5bd4b14d39b87f105702c613e5 [term 2 FOLLOWER]: Refusing update from remote peer f047b33669e648578b35878837bb0494: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:06:33.007546 15957 consensus_queue.cc:1035] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 14:06:33.013417 15758 raft_consensus.cc:1261] T c808defc7a8b4ecda4490809db5f983b P f047b33669e648578b35878837bb0494 [term 1 FOLLOWER]: Refusing update from remote peer ab642c5bd4b14d39b87f105702c613e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240502 14:06:33.013531 15723 consensus_peers.cc:485] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 -> Peer 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005): Couldn't send request to peer 17ee9f2434c74a02ad9a59074002db04. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:33.015312 15924 consensus_queue.cc:1035] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:33.016064 15957 raft_consensus.cc:2932] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 89449202af1248fb826e51c95e85d882 (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: true } } }
I20240502 14:06:33.017961 15886 raft_consensus.cc:1261] T c808defc7a8b4ecda4490809db5f983b P 89449202af1248fb826e51c95e85d882 [term 1 LEARNER]: Refusing update from remote peer ab642c5bd4b14d39b87f105702c613e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:06:33.021294 15950 consensus_queue.cc:1035] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20240502 14:06:33.023872 15649 consensus_peers.cc:485] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 -> Peer 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005): Couldn't send request to peer 17ee9f2434c74a02ad9a59074002db04. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:06:33.026635 15723 consensus_peers.cc:485] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 -> Peer 89449202af1248fb826e51c95e85d882 (127.11.9.68:45769): Couldn't send request to peer 89449202af1248fb826e51c95e85d882. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a727b7623ba4587bb183d2ce121ceed. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:33.027590 15683 raft_consensus.cc:2932] T 8a727b7623ba4587bb183d2ce121ceed P ab642c5bd4b14d39b87f105702c613e5 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 89449202af1248fb826e51c95e85d882 (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: true } } }
I20240502 14:06:33.029537 15491 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8a727b7623ba4587bb183d2ce121ceed with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:06:33.035210 15506 catalog_manager.cc:5484] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 reported cstate change: config changed from index -1 to 2, NON_VOTER 89449202af1248fb826e51c95e85d882 (127.11.9.68) added. New cstate: current_term: 2 leader_uuid: "f047b33669e648578b35878837bb0494" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:33.038797 15961 raft_consensus.cc:2932] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 89449202af1248fb826e51c95e85d882 (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } } }
I20240502 14:06:33.042251 15758 raft_consensus.cc:2932] T c808defc7a8b4ecda4490809db5f983b P f047b33669e648578b35878837bb0494 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 89449202af1248fb826e51c95e85d882 (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } } }
I20240502 14:06:33.052146 15886 raft_consensus.cc:2932] T c808defc7a8b4ecda4490809db5f983b P 89449202af1248fb826e51c95e85d882 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 89449202af1248fb826e51c95e85d882 (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } } }
I20240502 14:06:33.054283 15507 catalog_manager.cc:5484] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 reported cstate change: config changed from index 2 to 3, 89449202af1248fb826e51c95e85d882 (127.11.9.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab642c5bd4b14d39b87f105702c613e5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:06:33.057547 15555 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 17ee9f2434c74a02ad9a59074002db04 not available for placement
I20240502 14:06:33.065976 15683 consensus_queue.cc:237] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } }
I20240502 14:06:33.070134 15886 raft_consensus.cc:1261] T c808defc7a8b4ecda4490809db5f983b P 89449202af1248fb826e51c95e85d882 [term 1 FOLLOWER]: Refusing update from remote peer ab642c5bd4b14d39b87f105702c613e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:33.070856 15758 raft_consensus.cc:1261] T c808defc7a8b4ecda4490809db5f983b P f047b33669e648578b35878837bb0494 [term 1 FOLLOWER]: Refusing update from remote peer ab642c5bd4b14d39b87f105702c613e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:06:33.072135 15950 consensus_queue.cc:1035] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:33.073705 15950 consensus_queue.cc:1035] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:33.077667 15961 raft_consensus.cc:2932] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } } }
I20240502 14:06:33.079064 15886 raft_consensus.cc:2932] T c808defc7a8b4ecda4490809db5f983b P 89449202af1248fb826e51c95e85d882 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } } }
I20240502 14:06:33.081341 15758 raft_consensus.cc:2932] T c808defc7a8b4ecda4490809db5f983b P f047b33669e648578b35878837bb0494 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } } }
I20240502 14:06:33.086079 15493 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c808defc7a8b4ecda4490809db5f983b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:06:33.089270 15506 catalog_manager.cc:5484] T c808defc7a8b4ecda4490809db5f983b P 89449202af1248fb826e51c95e85d882 reported cstate change: config changed from index 3 to 4, VOTER 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65) evicted. New cstate: current_term: 1 leader_uuid: "ab642c5bd4b14d39b87f105702c613e5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } } }
W20240502 14:06:33.102470 15493 catalog_manager.cc:4573] TS 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c808defc7a8b4ecda4490809db5f983b: Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111)
I20240502 14:06:33.210861 15957 consensus_queue.cc:579] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [LEADER]: Leader has been unable to successfully communicate with peer 17ee9f2434c74a02ad9a59074002db04 for more than 1 seconds (1.050s)
I20240502 14:06:33.218890 15758 consensus_queue.cc:237] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: true } }
I20240502 14:06:33.223333 15683 raft_consensus.cc:1261] T dd54f15c1b87413c8541c924088c1cde P ab642c5bd4b14d39b87f105702c613e5 [term 2 FOLLOWER]: Refusing update from remote peer f047b33669e648578b35878837bb0494: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240502 14:06:33.224056 15723 consensus_peers.cc:485] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 -> Peer 89449202af1248fb826e51c95e85d882 (127.11.9.68:45769): Couldn't send request to peer 89449202af1248fb826e51c95e85d882. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd54f15c1b87413c8541c924088c1cde. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:33.225273 15934 consensus_queue.cc:1035] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240502 14:06:33.226395 15723 consensus_peers.cc:485] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 -> Peer 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005): Couldn't send request to peer 17ee9f2434c74a02ad9a59074002db04. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:33.230446 15923 raft_consensus.cc:2932] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 89449202af1248fb826e51c95e85d882 (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: true } } }
I20240502 14:06:33.231920 15683 raft_consensus.cc:2932] T dd54f15c1b87413c8541c924088c1cde P ab642c5bd4b14d39b87f105702c613e5 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 89449202af1248fb826e51c95e85d882 (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: true } } }
I20240502 14:06:33.238111 15491 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dd54f15c1b87413c8541c924088c1cde with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:06:33.242264 15507 catalog_manager.cc:5484] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 reported cstate change: config changed from index -1 to 2, NON_VOTER 89449202af1248fb826e51c95e85d882 (127.11.9.68) added. New cstate: current_term: 2 leader_uuid: "f047b33669e648578b35878837bb0494" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:33.270473 15967 ts_tablet_manager.cc:902] T 56ac0dbb1eef484bba6413e33b6729b3 P 89449202af1248fb826e51c95e85d882: Initiating tablet copy from peer ab642c5bd4b14d39b87f105702c613e5 (127.11.9.66:33825)
I20240502 14:06:33.273540 15967 tablet_copy_client.cc:308] T 56ac0dbb1eef484bba6413e33b6729b3 P 89449202af1248fb826e51c95e85d882: tablet copy: Beginning tablet copy session from remote peer at address 127.11.9.66:33825
I20240502 14:06:33.274906 15694 tablet_copy_service.cc:140] P ab642c5bd4b14d39b87f105702c613e5: Received BeginTabletCopySession request for tablet 56ac0dbb1eef484bba6413e33b6729b3 from peer 89449202af1248fb826e51c95e85d882 ({username='slave'} at 127.0.0.1:46764)
I20240502 14:06:33.275322 15694 tablet_copy_service.cc:161] P ab642c5bd4b14d39b87f105702c613e5: Beginning new tablet copy session on tablet 56ac0dbb1eef484bba6413e33b6729b3 from peer 89449202af1248fb826e51c95e85d882 at {username='slave'} at 127.0.0.1:46764: session id = 89449202af1248fb826e51c95e85d882-56ac0dbb1eef484bba6413e33b6729b3
I20240502 14:06:33.280149 15694 tablet_copy_source_session.cc:205] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:06:33.282779 15967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ac0dbb1eef484bba6413e33b6729b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:33.293761 15967 tablet_copy_client.cc:783] T 56ac0dbb1eef484bba6413e33b6729b3 P 89449202af1248fb826e51c95e85d882: tablet copy: Starting download of 0 data blocks...
I20240502 14:06:33.294193 15967 tablet_copy_client.cc:647] T 56ac0dbb1eef484bba6413e33b6729b3 P 89449202af1248fb826e51c95e85d882: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:33.297080 15967 tablet_copy_client.cc:523] T 56ac0dbb1eef484bba6413e33b6729b3 P 89449202af1248fb826e51c95e85d882: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:33.302433 15967 tablet_bootstrap.cc:492] T 56ac0dbb1eef484bba6413e33b6729b3 P 89449202af1248fb826e51c95e85d882: Bootstrap starting.
I20240502 14:06:33.315353 15967 tablet_bootstrap.cc:492] T 56ac0dbb1eef484bba6413e33b6729b3 P 89449202af1248fb826e51c95e85d882: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:33.316074 15967 tablet_bootstrap.cc:492] T 56ac0dbb1eef484bba6413e33b6729b3 P 89449202af1248fb826e51c95e85d882: Bootstrap complete.
I20240502 14:06:33.316567 15967 ts_tablet_manager.cc:1367] T 56ac0dbb1eef484bba6413e33b6729b3 P 89449202af1248fb826e51c95e85d882: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.003s
I20240502 14:06:33.318506 15967 raft_consensus.cc:348] T 56ac0dbb1eef484bba6413e33b6729b3 P 89449202af1248fb826e51c95e85d882 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: true } }
I20240502 14:06:33.319068 15967 raft_consensus.cc:729] T 56ac0dbb1eef484bba6413e33b6729b3 P 89449202af1248fb826e51c95e85d882 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 89449202af1248fb826e51c95e85d882, State: Initialized, Role: LEARNER
I20240502 14:06:33.319512 15967 consensus_queue.cc:260] T 56ac0dbb1eef484bba6413e33b6729b3 P 89449202af1248fb826e51c95e85d882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: true } }
I20240502 14:06:33.320959 15967 ts_tablet_manager.cc:1398] T 56ac0dbb1eef484bba6413e33b6729b3 P 89449202af1248fb826e51c95e85d882: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20240502 14:06:33.322338 15694 tablet_copy_service.cc:342] P ab642c5bd4b14d39b87f105702c613e5: Request end of tablet copy session 89449202af1248fb826e51c95e85d882-56ac0dbb1eef484bba6413e33b6729b3 received from {username='slave'} at 127.0.0.1:46764
I20240502 14:06:33.322633 15694 tablet_copy_service.cc:433] P ab642c5bd4b14d39b87f105702c613e5: ending tablet copy session 89449202af1248fb826e51c95e85d882-56ac0dbb1eef484bba6413e33b6729b3 on tablet 56ac0dbb1eef484bba6413e33b6729b3 with peer 89449202af1248fb826e51c95e85d882
I20240502 14:06:33.469100 15967 ts_tablet_manager.cc:902] T 8a727b7623ba4587bb183d2ce121ceed P 89449202af1248fb826e51c95e85d882: Initiating tablet copy from peer f047b33669e648578b35878837bb0494 (127.11.9.67:38309)
I20240502 14:06:33.472299 15967 tablet_copy_client.cc:308] T 8a727b7623ba4587bb183d2ce121ceed P 89449202af1248fb826e51c95e85d882: tablet copy: Beginning tablet copy session from remote peer at address 127.11.9.67:38309
I20240502 14:06:33.482064 15768 tablet_copy_service.cc:140] P f047b33669e648578b35878837bb0494: Received BeginTabletCopySession request for tablet 8a727b7623ba4587bb183d2ce121ceed from peer 89449202af1248fb826e51c95e85d882 ({username='slave'} at 127.0.0.1:38398)
I20240502 14:06:33.482502 15768 tablet_copy_service.cc:161] P f047b33669e648578b35878837bb0494: Beginning new tablet copy session on tablet 8a727b7623ba4587bb183d2ce121ceed from peer 89449202af1248fb826e51c95e85d882 at {username='slave'} at 127.0.0.1:38398: session id = 89449202af1248fb826e51c95e85d882-8a727b7623ba4587bb183d2ce121ceed
I20240502 14:06:33.487732 15768 tablet_copy_source_session.cc:205] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:06:33.490093 15967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a727b7623ba4587bb183d2ce121ceed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:33.499655 15967 tablet_copy_client.cc:783] T 8a727b7623ba4587bb183d2ce121ceed P 89449202af1248fb826e51c95e85d882: tablet copy: Starting download of 0 data blocks...
I20240502 14:06:33.500084 15967 tablet_copy_client.cc:647] T 8a727b7623ba4587bb183d2ce121ceed P 89449202af1248fb826e51c95e85d882: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:33.502916 15967 tablet_copy_client.cc:523] T 8a727b7623ba4587bb183d2ce121ceed P 89449202af1248fb826e51c95e85d882: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:33.508081 15967 tablet_bootstrap.cc:492] T 8a727b7623ba4587bb183d2ce121ceed P 89449202af1248fb826e51c95e85d882: Bootstrap starting.
I20240502 14:06:33.522465 15967 tablet_bootstrap.cc:492] T 8a727b7623ba4587bb183d2ce121ceed P 89449202af1248fb826e51c95e85d882: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:33.523094 15967 tablet_bootstrap.cc:492] T 8a727b7623ba4587bb183d2ce121ceed P 89449202af1248fb826e51c95e85d882: Bootstrap complete.
I20240502 14:06:33.523576 15967 ts_tablet_manager.cc:1367] T 8a727b7623ba4587bb183d2ce121ceed P 89449202af1248fb826e51c95e85d882: Time spent bootstrapping tablet: real 0.016s user 0.015s sys 0.000s
I20240502 14:06:33.525333 15967 raft_consensus.cc:348] T 8a727b7623ba4587bb183d2ce121ceed P 89449202af1248fb826e51c95e85d882 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: true } }
I20240502 14:06:33.525969 15967 raft_consensus.cc:729] T 8a727b7623ba4587bb183d2ce121ceed P 89449202af1248fb826e51c95e85d882 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 89449202af1248fb826e51c95e85d882, State: Initialized, Role: LEARNER
I20240502 14:06:33.526554 15967 consensus_queue.cc:260] T 8a727b7623ba4587bb183d2ce121ceed P 89449202af1248fb826e51c95e85d882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: true } }
I20240502 14:06:33.528705 15967 ts_tablet_manager.cc:1398] T 8a727b7623ba4587bb183d2ce121ceed P 89449202af1248fb826e51c95e85d882: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:06:33.530189 15768 tablet_copy_service.cc:342] P f047b33669e648578b35878837bb0494: Request end of tablet copy session 89449202af1248fb826e51c95e85d882-8a727b7623ba4587bb183d2ce121ceed received from {username='slave'} at 127.0.0.1:38398
I20240502 14:06:33.530637 15768 tablet_copy_service.cc:433] P f047b33669e648578b35878837bb0494: ending tablet copy session 89449202af1248fb826e51c95e85d882-8a727b7623ba4587bb183d2ce121ceed on tablet 8a727b7623ba4587bb183d2ce121ceed with peer 89449202af1248fb826e51c95e85d882
I20240502 14:06:33.766274 15967 ts_tablet_manager.cc:902] T dd54f15c1b87413c8541c924088c1cde P 89449202af1248fb826e51c95e85d882: Initiating tablet copy from peer f047b33669e648578b35878837bb0494 (127.11.9.67:38309)
I20240502 14:06:33.769270 15967 tablet_copy_client.cc:308] T dd54f15c1b87413c8541c924088c1cde P 89449202af1248fb826e51c95e85d882: tablet copy: Beginning tablet copy session from remote peer at address 127.11.9.67:38309
I20240502 14:06:33.770541 15768 tablet_copy_service.cc:140] P f047b33669e648578b35878837bb0494: Received BeginTabletCopySession request for tablet dd54f15c1b87413c8541c924088c1cde from peer 89449202af1248fb826e51c95e85d882 ({username='slave'} at 127.0.0.1:38398)
I20240502 14:06:33.770931 15768 tablet_copy_service.cc:161] P f047b33669e648578b35878837bb0494: Beginning new tablet copy session on tablet dd54f15c1b87413c8541c924088c1cde from peer 89449202af1248fb826e51c95e85d882 at {username='slave'} at 127.0.0.1:38398: session id = 89449202af1248fb826e51c95e85d882-dd54f15c1b87413c8541c924088c1cde
I20240502 14:06:33.775792 15768 tablet_copy_source_session.cc:205] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:06:33.778076 15967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd54f15c1b87413c8541c924088c1cde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:33.787603 15967 tablet_copy_client.cc:783] T dd54f15c1b87413c8541c924088c1cde P 89449202af1248fb826e51c95e85d882: tablet copy: Starting download of 0 data blocks...
I20240502 14:06:33.788048 15967 tablet_copy_client.cc:647] T dd54f15c1b87413c8541c924088c1cde P 89449202af1248fb826e51c95e85d882: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:33.791054 15967 tablet_copy_client.cc:523] T dd54f15c1b87413c8541c924088c1cde P 89449202af1248fb826e51c95e85d882: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:33.791533 15886 raft_consensus.cc:1203] T 56ac0dbb1eef484bba6413e33b6729b3 P 89449202af1248fb826e51c95e85d882 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20240502 14:06:33.797629 15967 tablet_bootstrap.cc:492] T dd54f15c1b87413c8541c924088c1cde P 89449202af1248fb826e51c95e85d882: Bootstrap starting.
I20240502 14:06:33.809006 15967 tablet_bootstrap.cc:492] T dd54f15c1b87413c8541c924088c1cde P 89449202af1248fb826e51c95e85d882: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:33.809648 15967 tablet_bootstrap.cc:492] T dd54f15c1b87413c8541c924088c1cde P 89449202af1248fb826e51c95e85d882: Bootstrap complete.
I20240502 14:06:33.810068 15967 ts_tablet_manager.cc:1367] T dd54f15c1b87413c8541c924088c1cde P 89449202af1248fb826e51c95e85d882: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.003s
I20240502 14:06:33.811578 15967 raft_consensus.cc:348] T dd54f15c1b87413c8541c924088c1cde P 89449202af1248fb826e51c95e85d882 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: true } }
I20240502 14:06:33.812084 15967 raft_consensus.cc:729] T dd54f15c1b87413c8541c924088c1cde P 89449202af1248fb826e51c95e85d882 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 89449202af1248fb826e51c95e85d882, State: Initialized, Role: LEARNER
I20240502 14:06:33.812469 15967 consensus_queue.cc:260] T dd54f15c1b87413c8541c924088c1cde P 89449202af1248fb826e51c95e85d882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: true } }
I20240502 14:06:33.813877 15967 ts_tablet_manager.cc:1398] T dd54f15c1b87413c8541c924088c1cde P 89449202af1248fb826e51c95e85d882: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 14:06:33.815095 15768 tablet_copy_service.cc:342] P f047b33669e648578b35878837bb0494: Request end of tablet copy session 89449202af1248fb826e51c95e85d882-dd54f15c1b87413c8541c924088c1cde received from {username='slave'} at 127.0.0.1:38398
I20240502 14:06:33.815405 15768 tablet_copy_service.cc:433] P f047b33669e648578b35878837bb0494: ending tablet copy session 89449202af1248fb826e51c95e85d882-dd54f15c1b87413c8541c924088c1cde on tablet dd54f15c1b87413c8541c924088c1cde with peer 89449202af1248fb826e51c95e85d882
I20240502 14:06:33.872805 15985 consensus_queue.cc:579] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [LEADER]: Leader has been unable to successfully communicate with peer 17ee9f2434c74a02ad9a59074002db04 for more than 1 seconds (1.103s)
I20240502 14:06:33.887476 15886 raft_consensus.cc:1203] T 8a727b7623ba4587bb183d2ce121ceed P 89449202af1248fb826e51c95e85d882 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20240502 14:06:34.043063 15990 consensus_queue.cc:579] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [LEADER]: Leader has been unable to successfully communicate with peer 17ee9f2434c74a02ad9a59074002db04 for more than 1 seconds (1.042s)
W20240502 14:06:34.059793 15555 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 17ee9f2434c74a02ad9a59074002db04 not available for placement
I20240502 14:06:34.221961 15990 consensus_queue.cc:579] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [LEADER]: Leader has been unable to successfully communicate with peer 17ee9f2434c74a02ad9a59074002db04 for more than 1 seconds (1.002s)
W20240502 14:06:34.223555 15493 catalog_manager.cc:4573] TS 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c808defc7a8b4ecda4490809db5f983b: Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111)
I20240502 14:06:34.319968 15886 raft_consensus.cc:1203] T dd54f15c1b87413c8541c924088c1cde P 89449202af1248fb826e51c95e85d882 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20240502 14:06:34.399237 15985 raft_consensus.cc:1050] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5: attempting to promote NON_VOTER 89449202af1248fb826e51c95e85d882 to VOTER
I20240502 14:06:34.401268 15985 consensus_queue.cc:237] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } }
I20240502 14:06:34.407052 15758 raft_consensus.cc:1261] T 56ac0dbb1eef484bba6413e33b6729b3 P f047b33669e648578b35878837bb0494 [term 2 FOLLOWER]: Refusing update from remote peer ab642c5bd4b14d39b87f105702c613e5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:06:34.408571 15985 consensus_queue.cc:1035] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:34.408624 15886 raft_consensus.cc:1261] T 56ac0dbb1eef484bba6413e33b6729b3 P 89449202af1248fb826e51c95e85d882 [term 2 LEARNER]: Refusing update from remote peer ab642c5bd4b14d39b87f105702c613e5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:06:34.410580 15985 consensus_queue.cc:1035] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 14:06:34.412179 15649 consensus_peers.cc:485] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 -> Peer 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005): Couldn't send request to peer 17ee9f2434c74a02ad9a59074002db04. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:34.417200 15995 raft_consensus.cc:2932] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 89449202af1248fb826e51c95e85d882 (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } } }
I20240502 14:06:34.418526 15886 raft_consensus.cc:2932] T 56ac0dbb1eef484bba6413e33b6729b3 P 89449202af1248fb826e51c95e85d882 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 89449202af1248fb826e51c95e85d882 (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } } }
I20240502 14:06:34.426386 15758 raft_consensus.cc:2932] T 56ac0dbb1eef484bba6413e33b6729b3 P f047b33669e648578b35878837bb0494 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 89449202af1248fb826e51c95e85d882 (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } } }
I20240502 14:06:34.427660 15507 catalog_manager.cc:5484] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 reported cstate change: config changed from index 2 to 3, 89449202af1248fb826e51c95e85d882 (127.11.9.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ab642c5bd4b14d39b87f105702c613e5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:06:34.442915 15683 consensus_queue.cc:237] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } }
I20240502 14:06:34.447089 15758 raft_consensus.cc:1261] T 56ac0dbb1eef484bba6413e33b6729b3 P f047b33669e648578b35878837bb0494 [term 2 FOLLOWER]: Refusing update from remote peer ab642c5bd4b14d39b87f105702c613e5: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:06:34.447089 15886 raft_consensus.cc:1261] T 56ac0dbb1eef484bba6413e33b6729b3 P 89449202af1248fb826e51c95e85d882 [term 2 FOLLOWER]: Refusing update from remote peer ab642c5bd4b14d39b87f105702c613e5: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:06:34.448485 15985 consensus_queue.cc:1035] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:34.449153 15995 consensus_queue.cc:1035] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:34.455101 15993 raft_consensus.cc:2932] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } } }
I20240502 14:06:34.456647 15886 raft_consensus.cc:2932] T 56ac0dbb1eef484bba6413e33b6729b3 P 89449202af1248fb826e51c95e85d882 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } } }
I20240502 14:06:34.457257 15758 raft_consensus.cc:2932] T 56ac0dbb1eef484bba6413e33b6729b3 P f047b33669e648578b35878837bb0494 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } } }
I20240502 14:06:34.462033 15493 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 56ac0dbb1eef484bba6413e33b6729b3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:06:34.465535 15506 catalog_manager.cc:5484] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 reported cstate change: config changed from index 3 to 4, VOTER 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65) evicted. New cstate: current_term: 2 leader_uuid: "ab642c5bd4b14d39b87f105702c613e5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:06:34.501211 15990 raft_consensus.cc:1050] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494: attempting to promote NON_VOTER 89449202af1248fb826e51c95e85d882 to VOTER
I20240502 14:06:34.502707 15990 consensus_queue.cc:237] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } }
I20240502 14:06:34.508277 15683 raft_consensus.cc:1261] T 8a727b7623ba4587bb183d2ce121ceed P ab642c5bd4b14d39b87f105702c613e5 [term 2 FOLLOWER]: Refusing update from remote peer f047b33669e648578b35878837bb0494: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:06:34.508483 15886 raft_consensus.cc:1261] T 8a727b7623ba4587bb183d2ce121ceed P 89449202af1248fb826e51c95e85d882 [term 2 LEARNER]: Refusing update from remote peer f047b33669e648578b35878837bb0494: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:06:34.509640 16003 consensus_queue.cc:1035] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:34.510583 15990 consensus_queue.cc:1035] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 14:06:34.511142 15723 consensus_peers.cc:485] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 -> Peer 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005): Couldn't send request to peer 17ee9f2434c74a02ad9a59074002db04. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:34.517196 15990 raft_consensus.cc:2932] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 89449202af1248fb826e51c95e85d882 (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } } }
I20240502 14:06:34.518659 15683 raft_consensus.cc:2932] T 8a727b7623ba4587bb183d2ce121ceed P ab642c5bd4b14d39b87f105702c613e5 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 89449202af1248fb826e51c95e85d882 (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } } }
I20240502 14:06:34.527302 15507 catalog_manager.cc:5484] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 reported cstate change: config changed from index 2 to 3, 89449202af1248fb826e51c95e85d882 (127.11.9.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f047b33669e648578b35878837bb0494" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:06:34.525288 15886 raft_consensus.cc:2932] T 8a727b7623ba4587bb183d2ce121ceed P 89449202af1248fb826e51c95e85d882 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 89449202af1248fb826e51c95e85d882 (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } } }
I20240502 14:06:34.537606 15758 consensus_queue.cc:237] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } }
I20240502 14:06:34.542593 15683 raft_consensus.cc:1261] T 8a727b7623ba4587bb183d2ce121ceed P ab642c5bd4b14d39b87f105702c613e5 [term 2 FOLLOWER]: Refusing update from remote peer f047b33669e648578b35878837bb0494: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:06:34.543037 15886 raft_consensus.cc:1261] T 8a727b7623ba4587bb183d2ce121ceed P 89449202af1248fb826e51c95e85d882 [term 2 FOLLOWER]: Refusing update from remote peer f047b33669e648578b35878837bb0494: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:06:34.543710 15990 consensus_queue.cc:1035] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:34.544488 15923 consensus_queue.cc:1035] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:34.549826 16003 raft_consensus.cc:2932] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } } }
I20240502 14:06:34.551720 15683 raft_consensus.cc:2932] T 8a727b7623ba4587bb183d2ce121ceed P ab642c5bd4b14d39b87f105702c613e5 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } } }
I20240502 14:06:34.551720 15886 raft_consensus.cc:2932] T 8a727b7623ba4587bb183d2ce121ceed P 89449202af1248fb826e51c95e85d882 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } } }
I20240502 14:06:34.556288 15491 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8a727b7623ba4587bb183d2ce121ceed with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:06:34.560150 15504 catalog_manager.cc:5484] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 reported cstate change: config changed from index 3 to 4, VOTER 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65) evicted. New cstate: current_term: 2 leader_uuid: "f047b33669e648578b35878837bb0494" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:06:34.795048 16003 raft_consensus.cc:1050] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494: attempting to promote NON_VOTER 89449202af1248fb826e51c95e85d882 to VOTER
I20240502 14:06:34.796736 16003 consensus_queue.cc:237] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } }
I20240502 14:06:34.801779 15683 raft_consensus.cc:1261] T dd54f15c1b87413c8541c924088c1cde P ab642c5bd4b14d39b87f105702c613e5 [term 2 FOLLOWER]: Refusing update from remote peer f047b33669e648578b35878837bb0494: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:06:34.802000 15886 raft_consensus.cc:1261] T dd54f15c1b87413c8541c924088c1cde P 89449202af1248fb826e51c95e85d882 [term 2 LEARNER]: Refusing update from remote peer f047b33669e648578b35878837bb0494: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:06:34.803227 15990 consensus_queue.cc:1035] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:34.803936 16003 consensus_queue.cc:1035] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 14:06:34.804889 15723 consensus_peers.cc:485] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 -> Peer 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005): Couldn't send request to peer 17ee9f2434c74a02ad9a59074002db04. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:34.811398 15990 raft_consensus.cc:2932] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 89449202af1248fb826e51c95e85d882 (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } } }
I20240502 14:06:34.812803 15886 raft_consensus.cc:2932] T dd54f15c1b87413c8541c924088c1cde P 89449202af1248fb826e51c95e85d882 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 89449202af1248fb826e51c95e85d882 (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } } }
I20240502 14:06:34.821849 15683 raft_consensus.cc:2932] T dd54f15c1b87413c8541c924088c1cde P ab642c5bd4b14d39b87f105702c613e5 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 89449202af1248fb826e51c95e85d882 (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } } }
I20240502 14:06:34.824822 15506 catalog_manager.cc:5484] T dd54f15c1b87413c8541c924088c1cde P 89449202af1248fb826e51c95e85d882 reported cstate change: config changed from index 2 to 3, 89449202af1248fb826e51c95e85d882 (127.11.9.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f047b33669e648578b35878837bb0494" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "17ee9f2434c74a02ad9a59074002db04" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 37005 } } peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } } }
I20240502 14:06:34.836681 15758 consensus_queue.cc:237] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } }
I20240502 14:06:34.840751 15886 raft_consensus.cc:1261] T dd54f15c1b87413c8541c924088c1cde P 89449202af1248fb826e51c95e85d882 [term 2 FOLLOWER]: Refusing update from remote peer f047b33669e648578b35878837bb0494: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:06:34.840849 15683 raft_consensus.cc:1261] T dd54f15c1b87413c8541c924088c1cde P ab642c5bd4b14d39b87f105702c613e5 [term 2 FOLLOWER]: Refusing update from remote peer f047b33669e648578b35878837bb0494: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:06:34.841814 15990 consensus_queue.cc:1035] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:34.842324 16003 consensus_queue.cc:1035] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:34.847492 15990 raft_consensus.cc:2932] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } } }
I20240502 14:06:34.849685 15683 raft_consensus.cc:2932] T dd54f15c1b87413c8541c924088c1cde P ab642c5bd4b14d39b87f105702c613e5 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } } }
I20240502 14:06:34.850322 15886 raft_consensus.cc:2932] T dd54f15c1b87413c8541c924088c1cde P 89449202af1248fb826e51c95e85d882 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } } }
I20240502 14:06:34.854532 15491 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dd54f15c1b87413c8541c924088c1cde with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:06:34.858481 15507 catalog_manager.cc:5484] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 reported cstate change: config changed from index 3 to 4, VOTER 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65) evicted. New cstate: current_term: 2 leader_uuid: "f047b33669e648578b35878837bb0494" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab642c5bd4b14d39b87f105702c613e5" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 33825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f047b33669e648578b35878837bb0494" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 38309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89449202af1248fb826e51c95e85d882" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 45769 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:06:35.229967 15493 catalog_manager.cc:4573] TS 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8a727b7623ba4587bb183d2ce121ceed: Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111)
W20240502 14:06:35.520817 15493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 14:06:36.779642 15493 catalog_manager.cc:4573] TS 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 56ac0dbb1eef484bba6413e33b6729b3: Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111)
I20240502 14:06:38.405025 11301 tablet_server.cc:178] TabletServer@127.11.9.66:0 shutting down...
I20240502 14:06:38.427637 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:06:38.428313 11301 tablet_replica.cc:330] T dd54f15c1b87413c8541c924088c1cde P ab642c5bd4b14d39b87f105702c613e5: stopping tablet replica
I20240502 14:06:38.428958 11301 raft_consensus.cc:2229] T dd54f15c1b87413c8541c924088c1cde P ab642c5bd4b14d39b87f105702c613e5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:38.429548 11301 raft_consensus.cc:2258] T dd54f15c1b87413c8541c924088c1cde P ab642c5bd4b14d39b87f105702c613e5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:38.431422 11301 tablet_replica.cc:330] T 8a727b7623ba4587bb183d2ce121ceed P ab642c5bd4b14d39b87f105702c613e5: stopping tablet replica
I20240502 14:06:38.431886 11301 raft_consensus.cc:2229] T 8a727b7623ba4587bb183d2ce121ceed P ab642c5bd4b14d39b87f105702c613e5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:38.432394 11301 raft_consensus.cc:2258] T 8a727b7623ba4587bb183d2ce121ceed P ab642c5bd4b14d39b87f105702c613e5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:38.434139 11301 tablet_replica.cc:330] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5: stopping tablet replica
I20240502 14:06:38.434595 11301 raft_consensus.cc:2229] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:38.435269 11301 raft_consensus.cc:2258] T c808defc7a8b4ecda4490809db5f983b P ab642c5bd4b14d39b87f105702c613e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:38.437186 11301 tablet_replica.cc:330] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5: stopping tablet replica
I20240502 14:06:38.437654 11301 raft_consensus.cc:2229] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [term 2 LEADER]: Raft consensus shutting down.
I20240502 14:06:38.438288 11301 raft_consensus.cc:2258] T 56ac0dbb1eef484bba6413e33b6729b3 P ab642c5bd4b14d39b87f105702c613e5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:38.460940 11301 tablet_server.cc:194] TabletServer@127.11.9.66:0 shutdown complete.
I20240502 14:06:38.474223 11301 tablet_server.cc:178] TabletServer@127.11.9.67:0 shutting down...
I20240502 14:06:38.501063 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:06:38.501657 11301 tablet_replica.cc:330] T 56ac0dbb1eef484bba6413e33b6729b3 P f047b33669e648578b35878837bb0494: stopping tablet replica
I20240502 14:06:38.502138 11301 raft_consensus.cc:2229] T 56ac0dbb1eef484bba6413e33b6729b3 P f047b33669e648578b35878837bb0494 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:38.502630 11301 raft_consensus.cc:2258] T 56ac0dbb1eef484bba6413e33b6729b3 P f047b33669e648578b35878837bb0494 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:38.504282 11301 tablet_replica.cc:330] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494: stopping tablet replica
I20240502 14:06:38.504776 11301 raft_consensus.cc:2229] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [term 2 LEADER]: Raft consensus shutting down.
I20240502 14:06:38.505437 11301 raft_consensus.cc:2258] T dd54f15c1b87413c8541c924088c1cde P f047b33669e648578b35878837bb0494 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:38.507149 11301 tablet_replica.cc:330] T c808defc7a8b4ecda4490809db5f983b P f047b33669e648578b35878837bb0494: stopping tablet replica
I20240502 14:06:38.507580 11301 raft_consensus.cc:2229] T c808defc7a8b4ecda4490809db5f983b P f047b33669e648578b35878837bb0494 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:38.508044 11301 raft_consensus.cc:2258] T c808defc7a8b4ecda4490809db5f983b P f047b33669e648578b35878837bb0494 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:38.509626 11301 tablet_replica.cc:330] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494: stopping tablet replica
I20240502 14:06:38.510031 11301 raft_consensus.cc:2229] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [term 2 LEADER]: Raft consensus shutting down.
I20240502 14:06:38.510681 11301 raft_consensus.cc:2258] T 8a727b7623ba4587bb183d2ce121ceed P f047b33669e648578b35878837bb0494 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:38.532291 11301 tablet_server.cc:194] TabletServer@127.11.9.67:0 shutdown complete.
I20240502 14:06:38.544447 11301 tablet_server.cc:178] TabletServer@127.11.9.68:0 shutting down...
I20240502 14:06:38.562233 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:06:38.562774 11301 tablet_replica.cc:330] T dd54f15c1b87413c8541c924088c1cde P 89449202af1248fb826e51c95e85d882: stopping tablet replica
I20240502 14:06:38.563264 11301 raft_consensus.cc:2229] T dd54f15c1b87413c8541c924088c1cde P 89449202af1248fb826e51c95e85d882 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:38.563733 11301 raft_consensus.cc:2258] T dd54f15c1b87413c8541c924088c1cde P 89449202af1248fb826e51c95e85d882 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:38.565397 11301 tablet_replica.cc:330] T 56ac0dbb1eef484bba6413e33b6729b3 P 89449202af1248fb826e51c95e85d882: stopping tablet replica
I20240502 14:06:38.565855 11301 raft_consensus.cc:2229] T 56ac0dbb1eef484bba6413e33b6729b3 P 89449202af1248fb826e51c95e85d882 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:38.566284 11301 raft_consensus.cc:2258] T 56ac0dbb1eef484bba6413e33b6729b3 P 89449202af1248fb826e51c95e85d882 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:38.567775 11301 tablet_replica.cc:330] T 8a727b7623ba4587bb183d2ce121ceed P 89449202af1248fb826e51c95e85d882: stopping tablet replica
I20240502 14:06:38.568202 11301 raft_consensus.cc:2229] T 8a727b7623ba4587bb183d2ce121ceed P 89449202af1248fb826e51c95e85d882 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:38.568614 11301 raft_consensus.cc:2258] T 8a727b7623ba4587bb183d2ce121ceed P 89449202af1248fb826e51c95e85d882 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:38.570062 11301 tablet_replica.cc:330] T c808defc7a8b4ecda4490809db5f983b P 89449202af1248fb826e51c95e85d882: stopping tablet replica
I20240502 14:06:38.570467 11301 raft_consensus.cc:2229] T c808defc7a8b4ecda4490809db5f983b P 89449202af1248fb826e51c95e85d882 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:38.570876 11301 raft_consensus.cc:2258] T c808defc7a8b4ecda4490809db5f983b P 89449202af1248fb826e51c95e85d882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:38.591274 11301 tablet_server.cc:194] TabletServer@127.11.9.68:0 shutdown complete.
I20240502 14:06:38.601429 11301 master.cc:537] Master@127.11.9.126:40531 shutting down...
W20240502 14:06:38.875396 15493 catalog_manager.cc:4573] TS 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8a727b7623ba4587bb183d2ce121ceed: Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111)
W20240502 14:06:41.481599 15493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:06:41.485194 15493 catalog_manager.cc:4573] TS 17ee9f2434c74a02ad9a59074002db04 (127.11.9.65:37005): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c808defc7a8b4ecda4490809db5f983b: Network error: Client connection negotiation failed: client connection to 127.11.9.65:37005: connect: Connection refused (error 111)
I20240502 14:06:41.503604 11301 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2b72fc549a36485daec3609edf942b24 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:41.504444 11301 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2b72fc549a36485daec3609edf942b24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:41.504812 11301 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2b72fc549a36485daec3609edf942b24: stopping tablet replica
I20240502 14:06:41.526677 11301 master.cc:552] Master@127.11.9.126:40531 shutdown complete.
[ OK ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering (13679 ms)
[ RUN ] AutoRebalancerTest.TestHandlingFailedTservers
I20240502 14:06:41.585853 11301 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.9.126:34149
I20240502 14:06:41.587008 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:41.592882 16016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:41.594334 16017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:41.594619 16019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:41.596766 11301 server_base.cc:1034] running on GCE node
I20240502 14:06:41.597738 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:06:41.597920 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:41.598052 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658801598035 us; error 0 us; skew 500 ppm
I20240502 14:06:41.598508 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:41.607793 11301 webserver.cc:460] Webserver started at http://127.11.9.126:42987/ using document root <none> and password file <none>
I20240502 14:06:41.608256 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:41.608428 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:41.608693 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:41.609967 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/master-0-root/instance:
uuid: "16a279f430ec4b5297e6c162adef8731"
format_stamp: "Formatted at 2024-05-02 14:06:41 on dist-test-slave-j72s"
I20240502 14:06:41.614243 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:06:41.617338 16024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:41.618023 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:06:41.618264 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/master-0-root
uuid: "16a279f430ec4b5297e6c162adef8731"
format_stamp: "Formatted at 2024-05-02 14:06:41 on dist-test-slave-j72s"
I20240502 14:06:41.618534 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:41.628707 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:41.629801 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:41.666926 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.126:34149
I20240502 14:06:41.667013 16075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.126:34149 every 8 connection(s)
I20240502 14:06:41.670922 16076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:41.682693 16076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16a279f430ec4b5297e6c162adef8731: Bootstrap starting.
I20240502 14:06:41.686686 16076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16a279f430ec4b5297e6c162adef8731: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:41.690476 16076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16a279f430ec4b5297e6c162adef8731: No bootstrap required, opened a new log
I20240502 14:06:41.692432 16076 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16a279f430ec4b5297e6c162adef8731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a279f430ec4b5297e6c162adef8731" member_type: VOTER }
I20240502 14:06:41.692893 16076 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16a279f430ec4b5297e6c162adef8731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:41.693162 16076 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16a279f430ec4b5297e6c162adef8731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a279f430ec4b5297e6c162adef8731, State: Initialized, Role: FOLLOWER
I20240502 14:06:41.693774 16076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16a279f430ec4b5297e6c162adef8731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a279f430ec4b5297e6c162adef8731" member_type: VOTER }
I20240502 14:06:41.694228 16076 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16a279f430ec4b5297e6c162adef8731 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:06:41.694429 16076 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16a279f430ec4b5297e6c162adef8731 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:06:41.694684 16076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16a279f430ec4b5297e6c162adef8731 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:41.698711 16076 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16a279f430ec4b5297e6c162adef8731 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a279f430ec4b5297e6c162adef8731" member_type: VOTER }
I20240502 14:06:41.699156 16076 leader_election.cc:302] T 00000000000000000000000000000000 P 16a279f430ec4b5297e6c162adef8731 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16a279f430ec4b5297e6c162adef8731; no voters:
I20240502 14:06:41.700167 16076 leader_election.cc:288] T 00000000000000000000000000000000 P 16a279f430ec4b5297e6c162adef8731 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:06:41.700416 16079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16a279f430ec4b5297e6c162adef8731 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:41.701754 16079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16a279f430ec4b5297e6c162adef8731 [term 1 LEADER]: Becoming Leader. State: Replica: 16a279f430ec4b5297e6c162adef8731, State: Running, Role: LEADER
I20240502 14:06:41.702406 16079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16a279f430ec4b5297e6c162adef8731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a279f430ec4b5297e6c162adef8731" member_type: VOTER }
I20240502 14:06:41.702973 16076 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16a279f430ec4b5297e6c162adef8731 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:06:41.705547 16081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a279f430ec4b5297e6c162adef8731 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16a279f430ec4b5297e6c162adef8731. Latest consensus state: current_term: 1 leader_uuid: "16a279f430ec4b5297e6c162adef8731" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a279f430ec4b5297e6c162adef8731" member_type: VOTER } }
I20240502 14:06:41.705570 16080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a279f430ec4b5297e6c162adef8731 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16a279f430ec4b5297e6c162adef8731" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a279f430ec4b5297e6c162adef8731" member_type: VOTER } }
I20240502 14:06:41.706962 16081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a279f430ec4b5297e6c162adef8731 [sys.catalog]: This master's current role is: LEADER
I20240502 14:06:41.707152 16080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a279f430ec4b5297e6c162adef8731 [sys.catalog]: This master's current role is: LEADER
I20240502 14:06:41.710633 16086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:06:41.718192 16086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:06:41.719188 11301 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:06:41.725876 16086 catalog_manager.cc:1347] Generated new cluster ID: a0bf03a72d8749588a06e2c190ded939
I20240502 14:06:41.726117 16086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:06:41.736557 16086 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:06:41.737805 16086 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:06:41.748348 16086 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 16a279f430ec4b5297e6c162adef8731: Generated new TSK 0
I20240502 14:06:41.748895 16086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:06:41.753017 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:41.758935 16097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:41.760365 16098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:41.761338 16100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:41.762477 11301 server_base.cc:1034] running on GCE node
I20240502 14:06:41.763227 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:06:41.763407 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:41.763540 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658801763526 us; error 0 us; skew 500 ppm
I20240502 14:06:41.764047 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:41.766304 11301 webserver.cc:460] Webserver started at http://127.11.9.65:39033/ using document root <none> and password file <none>
I20240502 14:06:41.766743 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:41.766918 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:41.767151 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:41.768358 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-0-root/instance:
uuid: "825bd1431f0845bbbc6fe2b99bd4df13"
format_stamp: "Formatted at 2024-05-02 14:06:41 on dist-test-slave-j72s"
I20240502 14:06:41.772578 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240502 14:06:41.775547 16105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:41.776189 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:06:41.776463 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-0-root
uuid: "825bd1431f0845bbbc6fe2b99bd4df13"
format_stamp: "Formatted at 2024-05-02 14:06:41 on dist-test-slave-j72s"
I20240502 14:06:41.776717 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:41.797116 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:41.798197 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:41.799563 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:41.801896 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:41.802062 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:41.802270 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:41.802465 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:41.841655 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.65:43963
I20240502 14:06:41.841738 16167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.65:43963 every 8 connection(s)
I20240502 14:06:41.846060 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:41.853886 16172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:41.854696 16173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:41.857648 16175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:41.858256 16168 heartbeater.cc:347] Connected to a master server at 127.11.9.126:34149
I20240502 14:06:41.858601 16168 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:41.858610 11301 server_base.cc:1034] running on GCE node
I20240502 14:06:41.859771 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 14:06:41.859767 16168 heartbeater.cc:511] Master 127.11.9.126:34149 requested a full tablet report, sending...
W20240502 14:06:41.860217 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:41.860433 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658801860414 us; error 0 us; skew 500 ppm
I20240502 14:06:41.861081 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:41.862143 16041 ts_manager.cc:194] Registered new tserver with Master: 825bd1431f0845bbbc6fe2b99bd4df13 (127.11.9.65:43963)
I20240502 14:06:41.863739 11301 webserver.cc:460] Webserver started at http://127.11.9.66:46113/ using document root <none> and password file <none>
I20240502 14:06:41.863911 16041 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44674
I20240502 14:06:41.864243 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:41.864470 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:41.864699 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:41.866254 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-1-root/instance:
uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30"
format_stamp: "Formatted at 2024-05-02 14:06:41 on dist-test-slave-j72s"
I20240502 14:06:41.870436 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20240502 14:06:41.873464 16180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:41.874161 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240502 14:06:41.874418 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-1-root
uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30"
format_stamp: "Formatted at 2024-05-02 14:06:41 on dist-test-slave-j72s"
I20240502 14:06:41.874663 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:41.884405 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:41.885403 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:41.886699 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:41.888774 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:41.888959 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:41.889156 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:41.889297 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:41.928516 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.66:42103
I20240502 14:06:41.928613 16242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.66:42103 every 8 connection(s)
I20240502 14:06:41.933204 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:41.940320 16246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:41.943349 16247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:41.944340 16249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:41.945050 11301 server_base.cc:1034] running on GCE node
I20240502 14:06:41.945655 16243 heartbeater.cc:347] Connected to a master server at 127.11.9.126:34149
I20240502 14:06:41.946074 16243 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:41.946043 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:06:41.946502 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:41.946713 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658801946697 us; error 0 us; skew 500 ppm
I20240502 14:06:41.947108 16243 heartbeater.cc:511] Master 127.11.9.126:34149 requested a full tablet report, sending...
I20240502 14:06:41.947521 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:41.949285 16041 ts_manager.cc:194] Registered new tserver with Master: 4aef7125ff234f1e9f4ab6ed3afa2b30 (127.11.9.66:42103)
I20240502 14:06:41.950488 11301 webserver.cc:460] Webserver started at http://127.11.9.67:46637/ using document root <none> and password file <none>
I20240502 14:06:41.951115 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:41.951236 16041 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44682
I20240502 14:06:41.951392 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:41.951771 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:41.953356 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-2-root/instance:
uuid: "591c3b60a468458eb2f5ba7cfccdc087"
format_stamp: "Formatted at 2024-05-02 14:06:41 on dist-test-slave-j72s"
I20240502 14:06:41.957685 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240502 14:06:41.960721 16254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:41.961431 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:06:41.961719 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-2-root
uuid: "591c3b60a468458eb2f5ba7cfccdc087"
format_stamp: "Formatted at 2024-05-02 14:06:41 on dist-test-slave-j72s"
I20240502 14:06:41.961975 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:41.976673 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:41.978168 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:41.981972 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:41.984436 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:41.984681 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:41.984961 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:41.985177 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:42.029349 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.67:37855
I20240502 14:06:42.029451 16316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.67:37855 every 8 connection(s)
I20240502 14:06:42.041985 16317 heartbeater.cc:347] Connected to a master server at 127.11.9.126:34149
I20240502 14:06:42.042312 16317 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:42.042991 16317 heartbeater.cc:511] Master 127.11.9.126:34149 requested a full tablet report, sending...
I20240502 14:06:42.044661 16041 ts_manager.cc:194] Registered new tserver with Master: 591c3b60a468458eb2f5ba7cfccdc087 (127.11.9.67:37855)
I20240502 14:06:42.045661 11301 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013207423s
I20240502 14:06:42.046000 16041 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44694
I20240502 14:06:42.866746 16168 heartbeater.cc:503] Master 127.11.9.126:34149 was elected leader, sending a full tablet report...
I20240502 14:06:42.954115 16243 heartbeater.cc:503] Master 127.11.9.126:34149 was elected leader, sending a full tablet report...
I20240502 14:06:43.048507 16317 heartbeater.cc:503] Master 127.11.9.126:34149 was elected leader, sending a full tablet report...
I20240502 14:06:43.077212 11301 test_util.cc:268] Using random seed: 484331590
I20240502 14:06:43.097875 16041 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44700:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:06:43.100085 16041 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:06:43.163514 16132 tablet_service.cc:1467] Processing CreateTablet for tablet 0d10c138d696478daff9fbf0a7571c45 (DEFAULT_TABLE table=test-workload [id=d0b54d7b2eed411486693626af79f4c3]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 14:06:43.164178 16131 tablet_service.cc:1467] Processing CreateTablet for tablet e2a87ab15cd7497abd4b275a559d7bba (DEFAULT_TABLE table=test-workload [id=d0b54d7b2eed411486693626af79f4c3]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 14:06:43.164837 16132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d10c138d696478daff9fbf0a7571c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:43.165433 16131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2a87ab15cd7497abd4b275a559d7bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:43.165505 16282 tablet_service.cc:1467] Processing CreateTablet for tablet 9ce141f797a744a4a44c6ade10abe773 (DEFAULT_TABLE table=test-workload [id=d0b54d7b2eed411486693626af79f4c3]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 14:06:43.173084 16205 tablet_service.cc:1467] Processing CreateTablet for tablet 473744e35b1a4da2ad3bac29156bcd9c (DEFAULT_TABLE table=test-workload [id=d0b54d7b2eed411486693626af79f4c3]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 14:06:43.165457 16130 tablet_service.cc:1467] Processing CreateTablet for tablet 473744e35b1a4da2ad3bac29156bcd9c (DEFAULT_TABLE table=test-workload [id=d0b54d7b2eed411486693626af79f4c3]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 14:06:43.174394 16205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 473744e35b1a4da2ad3bac29156bcd9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:43.174736 16130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 473744e35b1a4da2ad3bac29156bcd9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:43.166604 16281 tablet_service.cc:1467] Processing CreateTablet for tablet 0d10c138d696478daff9fbf0a7571c45 (DEFAULT_TABLE table=test-workload [id=d0b54d7b2eed411486693626af79f4c3]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 14:06:43.167786 16280 tablet_service.cc:1467] Processing CreateTablet for tablet e2a87ab15cd7497abd4b275a559d7bba (DEFAULT_TABLE table=test-workload [id=d0b54d7b2eed411486693626af79f4c3]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 14:06:43.168036 16208 tablet_service.cc:1467] Processing CreateTablet for tablet 9ce141f797a744a4a44c6ade10abe773 (DEFAULT_TABLE table=test-workload [id=d0b54d7b2eed411486693626af79f4c3]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 14:06:43.180182 16208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ce141f797a744a4a44c6ade10abe773. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:43.168987 16279 tablet_service.cc:1467] Processing CreateTablet for tablet 473744e35b1a4da2ad3bac29156bcd9c (DEFAULT_TABLE table=test-workload [id=d0b54d7b2eed411486693626af79f4c3]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 14:06:43.169723 16207 tablet_service.cc:1467] Processing CreateTablet for tablet 0d10c138d696478daff9fbf0a7571c45 (DEFAULT_TABLE table=test-workload [id=d0b54d7b2eed411486693626af79f4c3]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 14:06:43.171399 16206 tablet_service.cc:1467] Processing CreateTablet for tablet e2a87ab15cd7497abd4b275a559d7bba (DEFAULT_TABLE table=test-workload [id=d0b54d7b2eed411486693626af79f4c3]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 14:06:43.184036 16207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d10c138d696478daff9fbf0a7571c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:43.166509 16133 tablet_service.cc:1467] Processing CreateTablet for tablet 9ce141f797a744a4a44c6ade10abe773 (DEFAULT_TABLE table=test-workload [id=d0b54d7b2eed411486693626af79f4c3]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 14:06:43.185305 16133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ce141f797a744a4a44c6ade10abe773. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:43.190395 16206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2a87ab15cd7497abd4b275a559d7bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:43.190433 16337 tablet_bootstrap.cc:492] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13: Bootstrap starting.
I20240502 14:06:43.194008 16282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ce141f797a744a4a44c6ade10abe773. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:43.194595 16279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 473744e35b1a4da2ad3bac29156bcd9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:43.198772 16337 tablet_bootstrap.cc:654] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:43.202069 16280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2a87ab15cd7497abd4b275a559d7bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:43.203068 16338 tablet_bootstrap.cc:492] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30: Bootstrap starting.
I20240502 14:06:43.206506 16281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d10c138d696478daff9fbf0a7571c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:43.210196 16338 tablet_bootstrap.cc:654] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:43.238936 16337 tablet_bootstrap.cc:492] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13: No bootstrap required, opened a new log
I20240502 14:06:43.239588 16337 ts_tablet_manager.cc:1367] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13: Time spent bootstrapping tablet: real 0.049s user 0.014s sys 0.015s
I20240502 14:06:43.242874 16337 raft_consensus.cc:348] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.243695 16337 raft_consensus.cc:374] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:43.243950 16340 tablet_bootstrap.cc:492] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087: Bootstrap starting.
I20240502 14:06:43.244001 16337 raft_consensus.cc:729] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 825bd1431f0845bbbc6fe2b99bd4df13, State: Initialized, Role: FOLLOWER
I20240502 14:06:43.244936 16337 consensus_queue.cc:260] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.245577 16338 tablet_bootstrap.cc:492] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30: No bootstrap required, opened a new log
I20240502 14:06:43.246057 16338 ts_tablet_manager.cc:1367] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30: Time spent bootstrapping tablet: real 0.044s user 0.013s sys 0.000s
I20240502 14:06:43.247996 16337 ts_tablet_manager.cc:1398] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20240502 14:06:43.248875 16337 tablet_bootstrap.cc:492] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13: Bootstrap starting.
I20240502 14:06:43.248832 16338 raft_consensus.cc:348] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.249702 16338 raft_consensus.cc:374] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:43.249876 16340 tablet_bootstrap.cc:654] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:43.250013 16338 raft_consensus.cc:729] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aef7125ff234f1e9f4ab6ed3afa2b30, State: Initialized, Role: FOLLOWER
I20240502 14:06:43.250854 16338 consensus_queue.cc:260] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.255975 16337 tablet_bootstrap.cc:654] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:43.258558 16340 tablet_bootstrap.cc:492] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087: No bootstrap required, opened a new log
I20240502 14:06:43.258989 16340 ts_tablet_manager.cc:1367] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20240502 14:06:43.261550 16340 raft_consensus.cc:348] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.262285 16340 raft_consensus.cc:374] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:43.262557 16340 raft_consensus.cc:729] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591c3b60a468458eb2f5ba7cfccdc087, State: Initialized, Role: FOLLOWER
I20240502 14:06:43.263175 16340 consensus_queue.cc:260] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.266333 16338 ts_tablet_manager.cc:1398] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30: Time spent starting tablet: real 0.020s user 0.015s sys 0.005s
I20240502 14:06:43.267623 16338 tablet_bootstrap.cc:492] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30: Bootstrap starting.
I20240502 14:06:43.269562 16340 ts_tablet_manager.cc:1398] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s
I20240502 14:06:43.270457 16340 tablet_bootstrap.cc:492] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087: Bootstrap starting.
I20240502 14:06:43.275172 16338 tablet_bootstrap.cc:654] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:43.277220 16340 tablet_bootstrap.cc:654] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:43.277758 16337 tablet_bootstrap.cc:492] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13: No bootstrap required, opened a new log
I20240502 14:06:43.278331 16337 ts_tablet_manager.cc:1367] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.019s
I20240502 14:06:43.281232 16337 raft_consensus.cc:348] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.282147 16337 raft_consensus.cc:374] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:43.282573 16337 raft_consensus.cc:729] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 825bd1431f0845bbbc6fe2b99bd4df13, State: Initialized, Role: FOLLOWER
I20240502 14:06:43.283466 16337 consensus_queue.cc:260] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.287302 16337 ts_tablet_manager.cc:1398] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240502 14:06:43.288534 16337 tablet_bootstrap.cc:492] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13: Bootstrap starting.
I20240502 14:06:43.290505 16338 tablet_bootstrap.cc:492] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30: No bootstrap required, opened a new log
I20240502 14:06:43.291020 16338 ts_tablet_manager.cc:1367] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30: Time spent bootstrapping tablet: real 0.024s user 0.019s sys 0.000s
I20240502 14:06:43.290978 16340 tablet_bootstrap.cc:492] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087: No bootstrap required, opened a new log
I20240502 14:06:43.291921 16340 ts_tablet_manager.cc:1367] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087: Time spent bootstrapping tablet: real 0.022s user 0.015s sys 0.004s
I20240502 14:06:43.294373 16337 tablet_bootstrap.cc:654] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:43.294100 16338 raft_consensus.cc:348] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.294896 16338 raft_consensus.cc:374] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:43.295341 16338 raft_consensus.cc:729] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aef7125ff234f1e9f4ab6ed3afa2b30, State: Initialized, Role: FOLLOWER
I20240502 14:06:43.294770 16340 raft_consensus.cc:348] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.295928 16340 raft_consensus.cc:374] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:43.296342 16340 raft_consensus.cc:729] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591c3b60a468458eb2f5ba7cfccdc087, State: Initialized, Role: FOLLOWER
I20240502 14:06:43.296037 16338 consensus_queue.cc:260] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.297344 16340 consensus_queue.cc:260] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.298602 16338 ts_tablet_manager.cc:1398] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240502 14:06:43.299590 16338 tablet_bootstrap.cc:492] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30: Bootstrap starting.
I20240502 14:06:43.300436 16340 ts_tablet_manager.cc:1398] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20240502 14:06:43.301416 16340 tablet_bootstrap.cc:492] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087: Bootstrap starting.
I20240502 14:06:43.304991 16338 tablet_bootstrap.cc:654] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:43.307258 16340 tablet_bootstrap.cc:654] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:43.315168 16337 tablet_bootstrap.cc:492] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13: No bootstrap required, opened a new log
I20240502 14:06:43.315739 16337 ts_tablet_manager.cc:1367] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.007s
I20240502 14:06:43.318989 16337 raft_consensus.cc:348] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.319936 16337 raft_consensus.cc:374] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:43.320287 16337 raft_consensus.cc:729] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 825bd1431f0845bbbc6fe2b99bd4df13, State: Initialized, Role: FOLLOWER
I20240502 14:06:43.321583 16337 consensus_queue.cc:260] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.324514 16337 ts_tablet_manager.cc:1398] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20240502 14:06:43.325446 16337 tablet_bootstrap.cc:492] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13: Bootstrap starting.
I20240502 14:06:43.327196 16338 tablet_bootstrap.cc:492] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30: No bootstrap required, opened a new log
I20240502 14:06:43.327605 16338 ts_tablet_manager.cc:1367] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30: Time spent bootstrapping tablet: real 0.028s user 0.015s sys 0.011s
I20240502 14:06:43.329871 16338 raft_consensus.cc:348] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.330580 16338 raft_consensus.cc:374] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:43.330718 16340 tablet_bootstrap.cc:492] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087: No bootstrap required, opened a new log
I20240502 14:06:43.330910 16338 raft_consensus.cc:729] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aef7125ff234f1e9f4ab6ed3afa2b30, State: Initialized, Role: FOLLOWER
I20240502 14:06:43.331274 16340 ts_tablet_manager.cc:1367] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.015s
I20240502 14:06:43.331322 16337 tablet_bootstrap.cc:654] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:43.331741 16338 consensus_queue.cc:260] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.333673 16342 raft_consensus.cc:482] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:43.334787 16346 raft_consensus.cc:482] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:43.334137 16342 raft_consensus.cc:504] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.335269 16346 raft_consensus.cc:504] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.335670 16338 ts_tablet_manager.cc:1398] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240502 14:06:43.335669 16340 raft_consensus.cc:348] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.337061 16340 raft_consensus.cc:374] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:43.337452 16340 raft_consensus.cc:729] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591c3b60a468458eb2f5ba7cfccdc087, State: Initialized, Role: FOLLOWER
I20240502 14:06:43.337229 16338 tablet_bootstrap.cc:492] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30: Bootstrap starting.
I20240502 14:06:43.338359 16340 consensus_queue.cc:260] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.343950 16338 tablet_bootstrap.cc:654] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:43.345851 16342 leader_election.cc:288] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aef7125ff234f1e9f4ab6ed3afa2b30 (127.11.9.66:42103), 591c3b60a468458eb2f5ba7cfccdc087 (127.11.9.67:37855)
I20240502 14:06:43.350569 16346 leader_election.cc:288] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aef7125ff234f1e9f4ab6ed3afa2b30 (127.11.9.66:42103), 825bd1431f0845bbbc6fe2b99bd4df13 (127.11.9.65:43963)
I20240502 14:06:43.370884 16338 tablet_bootstrap.cc:492] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30: No bootstrap required, opened a new log
I20240502 14:06:43.371443 16338 ts_tablet_manager.cc:1367] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.010s
I20240502 14:06:43.377467 16337 tablet_bootstrap.cc:492] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13: No bootstrap required, opened a new log
I20240502 14:06:43.377080 16338 raft_consensus.cc:348] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.378033 16337 ts_tablet_manager.cc:1367] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13: Time spent bootstrapping tablet: real 0.053s user 0.021s sys 0.023s
I20240502 14:06:43.378067 16338 raft_consensus.cc:374] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:43.378647 16338 raft_consensus.cc:729] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aef7125ff234f1e9f4ab6ed3afa2b30, State: Initialized, Role: FOLLOWER
I20240502 14:06:43.379526 16338 consensus_queue.cc:260] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.380973 16337 raft_consensus.cc:348] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.381693 16338 ts_tablet_manager.cc:1398] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20240502 14:06:43.381716 16337 raft_consensus.cc:374] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:43.382225 16337 raft_consensus.cc:729] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 825bd1431f0845bbbc6fe2b99bd4df13, State: Initialized, Role: FOLLOWER
I20240502 14:06:43.382891 16337 consensus_queue.cc:260] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.384260 16340 ts_tablet_manager.cc:1398] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087: Time spent starting tablet: real 0.053s user 0.013s sys 0.008s
I20240502 14:06:43.385066 16337 ts_tablet_manager.cc:1398] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240502 14:06:43.389709 16143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce141f797a744a4a44c6ade10abe773" candidate_uuid: "591c3b60a468458eb2f5ba7cfccdc087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" is_pre_election: true
I20240502 14:06:43.390545 16143 raft_consensus.cc:2454] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 591c3b60a468458eb2f5ba7cfccdc087 in term 0.
I20240502 14:06:43.391500 16340 tablet_bootstrap.cc:492] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087: Bootstrap starting.
I20240502 14:06:43.392400 16218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "473744e35b1a4da2ad3bac29156bcd9c" candidate_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" is_pre_election: true
I20240502 14:06:43.393078 16218 raft_consensus.cc:2454] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 825bd1431f0845bbbc6fe2b99bd4df13 in term 0.
I20240502 14:06:43.395179 16258 leader_election.cc:302] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 591c3b60a468458eb2f5ba7cfccdc087, 825bd1431f0845bbbc6fe2b99bd4df13; no voters:
I20240502 14:06:43.395537 16109 leader_election.cc:302] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aef7125ff234f1e9f4ab6ed3afa2b30, 825bd1431f0845bbbc6fe2b99bd4df13; no voters:
I20240502 14:06:43.396219 16355 raft_consensus.cc:2781] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:43.396443 16342 raft_consensus.cc:2781] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:43.396682 16355 raft_consensus.cc:482] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:43.396868 16342 raft_consensus.cc:482] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:43.397070 16355 raft_consensus.cc:3037] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:43.397234 16342 raft_consensus.cc:3037] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:43.397461 16340 tablet_bootstrap.cc:654] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:43.402295 16340 tablet_bootstrap.cc:492] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087: No bootstrap required, opened a new log
I20240502 14:06:43.402501 16292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "473744e35b1a4da2ad3bac29156bcd9c" candidate_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "591c3b60a468458eb2f5ba7cfccdc087" is_pre_election: true
I20240502 14:06:43.403215 16292 raft_consensus.cc:2454] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 825bd1431f0845bbbc6fe2b99bd4df13 in term 0.
I20240502 14:06:43.404444 16342 raft_consensus.cc:504] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.407150 16218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "473744e35b1a4da2ad3bac29156bcd9c" candidate_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30"
I20240502 14:06:43.406898 16355 raft_consensus.cc:504] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.407781 16218 raft_consensus.cc:3037] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:43.402758 16340 ts_tablet_manager.cc:1367] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240502 14:06:43.409219 16342 leader_election.cc:288] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [CANDIDATE]: Term 1 election: Requested vote from peers 4aef7125ff234f1e9f4ab6ed3afa2b30 (127.11.9.66:42103), 591c3b60a468458eb2f5ba7cfccdc087 (127.11.9.67:37855)
I20240502 14:06:43.410025 16217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce141f797a744a4a44c6ade10abe773" candidate_uuid: "591c3b60a468458eb2f5ba7cfccdc087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" is_pre_election: true
I20240502 14:06:43.410274 16292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "473744e35b1a4da2ad3bac29156bcd9c" candidate_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "591c3b60a468458eb2f5ba7cfccdc087"
I20240502 14:06:43.410604 16217 raft_consensus.cc:2454] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 591c3b60a468458eb2f5ba7cfccdc087 in term 0.
I20240502 14:06:43.410878 16292 raft_consensus.cc:3037] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:43.411556 16216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce141f797a744a4a44c6ade10abe773" candidate_uuid: "591c3b60a468458eb2f5ba7cfccdc087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30"
I20240502 14:06:43.412134 16216 raft_consensus.cc:3037] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:43.413215 16355 leader_election.cc:288] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [CANDIDATE]: Term 1 election: Requested vote from peers 4aef7125ff234f1e9f4ab6ed3afa2b30 (127.11.9.66:42103), 825bd1431f0845bbbc6fe2b99bd4df13 (127.11.9.65:43963)
I20240502 14:06:43.412164 16340 raft_consensus.cc:348] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.413975 16340 raft_consensus.cc:374] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:43.414268 16340 raft_consensus.cc:729] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591c3b60a468458eb2f5ba7cfccdc087, State: Initialized, Role: FOLLOWER
I20240502 14:06:43.414332 16143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce141f797a744a4a44c6ade10abe773" candidate_uuid: "591c3b60a468458eb2f5ba7cfccdc087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "825bd1431f0845bbbc6fe2b99bd4df13"
I20240502 14:06:43.414978 16143 raft_consensus.cc:3037] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:43.415122 16340 consensus_queue.cc:260] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.418035 16292 raft_consensus.cc:2454] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 825bd1431f0845bbbc6fe2b99bd4df13 in term 1.
I20240502 14:06:43.419235 16106 leader_election.cc:302] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 591c3b60a468458eb2f5ba7cfccdc087, 825bd1431f0845bbbc6fe2b99bd4df13; no voters:
I20240502 14:06:43.420233 16342 raft_consensus.cc:2781] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:43.420321 16340 ts_tablet_manager.cc:1398] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087: Time spent starting tablet: real 0.011s user 0.004s sys 0.003s
I20240502 14:06:43.421759 16342 raft_consensus.cc:686] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 LEADER]: Becoming Leader. State: Replica: 825bd1431f0845bbbc6fe2b99bd4df13, State: Running, Role: LEADER
I20240502 14:06:43.421859 16143 raft_consensus.cc:2454] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 591c3b60a468458eb2f5ba7cfccdc087 in term 1.
I20240502 14:06:43.422549 16342 consensus_queue.cc:237] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.423552 16258 leader_election.cc:302] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 591c3b60a468458eb2f5ba7cfccdc087, 825bd1431f0845bbbc6fe2b99bd4df13; no voters:
I20240502 14:06:43.424329 16355 raft_consensus.cc:2781] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:43.424774 16355 raft_consensus.cc:686] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 LEADER]: Becoming Leader. State: Replica: 591c3b60a468458eb2f5ba7cfccdc087, State: Running, Role: LEADER
I20240502 14:06:43.425714 16355 consensus_queue.cc:237] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.426810 16218 raft_consensus.cc:2454] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 825bd1431f0845bbbc6fe2b99bd4df13 in term 1.
I20240502 14:06:43.426918 16216 raft_consensus.cc:2454] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 591c3b60a468458eb2f5ba7cfccdc087 in term 1.
I20240502 14:06:43.436044 16038 catalog_manager.cc:5484] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 reported cstate change: term changed from 0 to 1, leader changed from <none> to 825bd1431f0845bbbc6fe2b99bd4df13 (127.11.9.65). New cstate: current_term: 1 leader_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:43.438107 16039 catalog_manager.cc:5484] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 reported cstate change: term changed from 0 to 1, leader changed from <none> to 591c3b60a468458eb2f5ba7cfccdc087 (127.11.9.67). New cstate: current_term: 1 leader_uuid: "591c3b60a468458eb2f5ba7cfccdc087" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } health_report { overall_health: HEALTHY } } }
I20240502 14:06:43.456105 16355 raft_consensus.cc:482] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:43.456565 16355 raft_consensus.cc:504] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.458043 16355 leader_election.cc:288] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 825bd1431f0845bbbc6fe2b99bd4df13 (127.11.9.65:43963), 4aef7125ff234f1e9f4ab6ed3afa2b30 (127.11.9.66:42103)
I20240502 14:06:43.458782 16143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2a87ab15cd7497abd4b275a559d7bba" candidate_uuid: "591c3b60a468458eb2f5ba7cfccdc087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" is_pre_election: true
I20240502 14:06:43.458933 16218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2a87ab15cd7497abd4b275a559d7bba" candidate_uuid: "591c3b60a468458eb2f5ba7cfccdc087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" is_pre_election: true
I20240502 14:06:43.459297 16143 raft_consensus.cc:2454] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 591c3b60a468458eb2f5ba7cfccdc087 in term 0.
I20240502 14:06:43.459481 16218 raft_consensus.cc:2454] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 591c3b60a468458eb2f5ba7cfccdc087 in term 0.
I20240502 14:06:43.460152 16258 leader_election.cc:302] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 591c3b60a468458eb2f5ba7cfccdc087, 825bd1431f0845bbbc6fe2b99bd4df13; no voters:
I20240502 14:06:43.460726 16355 raft_consensus.cc:2781] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:43.460986 16355 raft_consensus.cc:482] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:43.461248 16355 raft_consensus.cc:3037] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:43.465714 16355 raft_consensus.cc:504] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.467095 16355 leader_election.cc:288] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [CANDIDATE]: Term 1 election: Requested vote from peers 825bd1431f0845bbbc6fe2b99bd4df13 (127.11.9.65:43963), 4aef7125ff234f1e9f4ab6ed3afa2b30 (127.11.9.66:42103)
I20240502 14:06:43.467916 16143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2a87ab15cd7497abd4b275a559d7bba" candidate_uuid: "591c3b60a468458eb2f5ba7cfccdc087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "825bd1431f0845bbbc6fe2b99bd4df13"
I20240502 14:06:43.468092 16218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2a87ab15cd7497abd4b275a559d7bba" candidate_uuid: "591c3b60a468458eb2f5ba7cfccdc087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30"
I20240502 14:06:43.468519 16143 raft_consensus.cc:3037] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:43.468613 16218 raft_consensus.cc:3037] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:43.474897 16143 raft_consensus.cc:2454] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 591c3b60a468458eb2f5ba7cfccdc087 in term 1.
I20240502 14:06:43.474897 16218 raft_consensus.cc:2454] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 591c3b60a468458eb2f5ba7cfccdc087 in term 1.
I20240502 14:06:43.475921 16258 leader_election.cc:302] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aef7125ff234f1e9f4ab6ed3afa2b30, 591c3b60a468458eb2f5ba7cfccdc087; no voters:
I20240502 14:06:43.476461 16355 raft_consensus.cc:2781] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:43.476822 16355 raft_consensus.cc:686] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 LEADER]: Becoming Leader. State: Replica: 591c3b60a468458eb2f5ba7cfccdc087, State: Running, Role: LEADER
I20240502 14:06:43.477595 16355 consensus_queue.cc:237] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.483659 16039 catalog_manager.cc:5484] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 reported cstate change: term changed from 0 to 1, leader changed from <none> to 591c3b60a468458eb2f5ba7cfccdc087 (127.11.9.67). New cstate: current_term: 1 leader_uuid: "591c3b60a468458eb2f5ba7cfccdc087" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } health_report { overall_health: HEALTHY } } }
I20240502 14:06:43.526655 16355 raft_consensus.cc:482] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:43.527113 16355 raft_consensus.cc:504] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.528591 16355 leader_election.cc:288] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 825bd1431f0845bbbc6fe2b99bd4df13 (127.11.9.65:43963), 4aef7125ff234f1e9f4ab6ed3afa2b30 (127.11.9.66:42103)
I20240502 14:06:43.529381 16143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d10c138d696478daff9fbf0a7571c45" candidate_uuid: "591c3b60a468458eb2f5ba7cfccdc087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" is_pre_election: true
I20240502 14:06:43.529624 16218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d10c138d696478daff9fbf0a7571c45" candidate_uuid: "591c3b60a468458eb2f5ba7cfccdc087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" is_pre_election: true
I20240502 14:06:43.530103 16143 raft_consensus.cc:2454] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 591c3b60a468458eb2f5ba7cfccdc087 in term 0.
I20240502 14:06:43.530160 16218 raft_consensus.cc:2454] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 591c3b60a468458eb2f5ba7cfccdc087 in term 0.
I20240502 14:06:43.531051 16258 leader_election.cc:302] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aef7125ff234f1e9f4ab6ed3afa2b30, 591c3b60a468458eb2f5ba7cfccdc087; no voters:
I20240502 14:06:43.531544 16355 raft_consensus.cc:2781] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:43.531805 16355 raft_consensus.cc:482] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:43.532044 16355 raft_consensus.cc:3037] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:43.537130 16355 raft_consensus.cc:504] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.538431 16355 leader_election.cc:288] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [CANDIDATE]: Term 1 election: Requested vote from peers 825bd1431f0845bbbc6fe2b99bd4df13 (127.11.9.65:43963), 4aef7125ff234f1e9f4ab6ed3afa2b30 (127.11.9.66:42103)
I20240502 14:06:43.539160 16143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d10c138d696478daff9fbf0a7571c45" candidate_uuid: "591c3b60a468458eb2f5ba7cfccdc087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "825bd1431f0845bbbc6fe2b99bd4df13"
I20240502 14:06:43.539330 16218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d10c138d696478daff9fbf0a7571c45" candidate_uuid: "591c3b60a468458eb2f5ba7cfccdc087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30"
I20240502 14:06:43.539712 16143 raft_consensus.cc:3037] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:43.539800 16218 raft_consensus.cc:3037] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:43.545408 16143 raft_consensus.cc:2454] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 591c3b60a468458eb2f5ba7cfccdc087 in term 1.
I20240502 14:06:43.545411 16218 raft_consensus.cc:2454] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 591c3b60a468458eb2f5ba7cfccdc087 in term 1.
I20240502 14:06:43.546483 16258 leader_election.cc:302] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aef7125ff234f1e9f4ab6ed3afa2b30, 591c3b60a468458eb2f5ba7cfccdc087; no voters:
I20240502 14:06:43.547047 16355 raft_consensus.cc:2781] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:43.547410 16355 raft_consensus.cc:686] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 LEADER]: Becoming Leader. State: Replica: 591c3b60a468458eb2f5ba7cfccdc087, State: Running, Role: LEADER
I20240502 14:06:43.548090 16355 consensus_queue.cc:237] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:43.553972 16039 catalog_manager.cc:5484] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 reported cstate change: term changed from 0 to 1, leader changed from <none> to 591c3b60a468458eb2f5ba7cfccdc087 (127.11.9.67). New cstate: current_term: 1 leader_uuid: "591c3b60a468458eb2f5ba7cfccdc087" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } health_report { overall_health: HEALTHY } } }
I20240502 14:06:43.561513 11301 tablet_server.cc:178] TabletServer@127.11.9.65:0 shutting down...
I20240502 14:06:43.580272 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:06:43.580868 11301 tablet_replica.cc:330] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13: stopping tablet replica
I20240502 14:06:43.581432 11301 raft_consensus.cc:2229] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:43.582017 11301 pending_rounds.cc:62] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13: Trying to abort 1 pending ops.
I20240502 14:06:43.582212 11301 pending_rounds.cc:68] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023242458827296768 op_type: NO_OP noop_request { }
I20240502 14:06:43.582484 11301 raft_consensus.cc:2866] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 14:06:43.582720 11301 raft_consensus.cc:2258] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:43.584690 11301 tablet_replica.cc:330] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13: stopping tablet replica
I20240502 14:06:43.585119 11301 raft_consensus.cc:2229] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:43.585419 11301 raft_consensus.cc:2258] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:43.586956 11301 tablet_replica.cc:330] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13: stopping tablet replica
I20240502 14:06:43.587409 11301 raft_consensus.cc:2229] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:43.587740 11301 raft_consensus.cc:2258] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:43.589263 11301 tablet_replica.cc:330] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13: stopping tablet replica
I20240502 14:06:43.589716 11301 raft_consensus.cc:2229] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:43.590016 11301 raft_consensus.cc:2258] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:43.611775 11301 tablet_server.cc:194] TabletServer@127.11.9.65:0 shutdown complete.
I20240502 14:06:43.621078 11301 tablet_server.cc:178] TabletServer@127.11.9.66:0 shutting down...
I20240502 14:06:43.636083 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:06:43.636790 11301 tablet_replica.cc:330] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30: stopping tablet replica
I20240502 14:06:43.637353 11301 raft_consensus.cc:2229] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:43.637737 11301 raft_consensus.cc:2258] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:43.639336 11301 tablet_replica.cc:330] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30: stopping tablet replica
I20240502 14:06:43.639798 11301 raft_consensus.cc:2229] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:43.640111 11301 raft_consensus.cc:2258] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:43.641693 11301 tablet_replica.cc:330] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30: stopping tablet replica
I20240502 14:06:43.642122 11301 raft_consensus.cc:2229] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:43.642434 11301 raft_consensus.cc:2258] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:43.643908 11301 tablet_replica.cc:330] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30: stopping tablet replica
I20240502 14:06:43.644309 11301 raft_consensus.cc:2229] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:43.644619 11301 raft_consensus.cc:2258] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:43.665241 11301 tablet_server.cc:194] TabletServer@127.11.9.66:0 shutdown complete.
I20240502 14:06:43.674713 11301 tablet_server.cc:178] TabletServer@127.11.9.67:0 shutting down...
I20240502 14:06:43.690636 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:06:43.691382 11301 tablet_replica.cc:330] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087: stopping tablet replica
I20240502 14:06:43.691854 11301 raft_consensus.cc:2229] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:43.692232 11301 raft_consensus.cc:2258] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:43.693961 11301 tablet_replica.cc:330] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087: stopping tablet replica
I20240502 14:06:43.694412 11301 raft_consensus.cc:2229] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:43.694892 11301 pending_rounds.cc:62] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087: Trying to abort 1 pending ops.
I20240502 14:06:43.695052 11301 pending_rounds.cc:68] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023242459338911744 op_type: NO_OP noop_request { }
I20240502 14:06:43.695304 11301 raft_consensus.cc:2866] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 14:06:43.695547 11301 raft_consensus.cc:2258] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:43.697645 11301 tablet_replica.cc:330] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087: stopping tablet replica
I20240502 14:06:43.698073 11301 raft_consensus.cc:2229] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:43.698554 11301 pending_rounds.cc:62] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087: Trying to abort 1 pending ops.
I20240502 14:06:43.698702 11301 pending_rounds.cc:68] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023242459050500096 op_type: NO_OP noop_request { }
I20240502 14:06:43.698942 11301 raft_consensus.cc:2866] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 14:06:43.699174 11301 raft_consensus.cc:2258] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:43.701117 11301 tablet_replica.cc:330] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087: stopping tablet replica
I20240502 14:06:43.701587 11301 raft_consensus.cc:2229] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:43.702044 11301 pending_rounds.cc:62] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087: Trying to abort 1 pending ops.
I20240502 14:06:43.702178 11301 pending_rounds.cc:68] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023242458840367104 op_type: NO_OP noop_request { }
I20240502 14:06:43.702431 11301 raft_consensus.cc:2866] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 14:06:43.702661 11301 raft_consensus.cc:2258] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:43.724494 11301 tablet_server.cc:194] TabletServer@127.11.9.67:0 shutdown complete.
I20240502 14:06:43.734817 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:43.740136 16366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:43.742413 16367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:43.742543 16369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:43.743234 11301 server_base.cc:1034] running on GCE node
I20240502 14:06:43.744190 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:06:43.744484 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:43.744721 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658803744708 us; error 0 us; skew 500 ppm
I20240502 14:06:43.745292 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:43.747696 11301 webserver.cc:460] Webserver started at http://127.11.9.68:37801/ using document root <none> and password file <none>
I20240502 14:06:43.748200 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:43.748461 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:43.748769 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:43.749905 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-3-root/instance:
uuid: "79a2dede1857429e96fe526b406a2b88"
format_stamp: "Formatted at 2024-05-02 14:06:43 on dist-test-slave-j72s"
I20240502 14:06:43.754142 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 14:06:43.757140 16374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:43.757982 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:06:43.758335 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-3-root
uuid: "79a2dede1857429e96fe526b406a2b88"
format_stamp: "Formatted at 2024-05-02 14:06:43 on dist-test-slave-j72s"
I20240502 14:06:43.758706 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:43.782505 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:43.783579 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:43.785019 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:43.787285 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:43.787598 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:43.787878 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:43.788111 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:43.823782 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.68:39969
I20240502 14:06:43.823873 16436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.68:39969 every 8 connection(s)
I20240502 14:06:43.837116 16437 heartbeater.cc:347] Connected to a master server at 127.11.9.126:34149
I20240502 14:06:43.837581 16437 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:43.838299 16437 heartbeater.cc:511] Master 127.11.9.126:34149 requested a full tablet report, sending...
I20240502 14:06:43.840044 16038 ts_manager.cc:194] Registered new tserver with Master: 79a2dede1857429e96fe526b406a2b88 (127.11.9.68:39969)
I20240502 14:06:43.841423 16038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44706
W20240502 14:06:44.729683 16089 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.11.9.67:37855: connect: Connection refused (error 111)
W20240502 14:06:44.734674 16089 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.11.9.67:37855: connect: Connection refused (error 111)
W20240502 14:06:44.734980 16089 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.11.9.67:37855: connect: Connection refused (error 111)
W20240502 14:06:44.738586 16089 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.11.9.65:43963: connect: Connection refused (error 111)
W20240502 14:06:44.738888 16089 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.11.9.65:43963: connect: Connection refused (error 111)
W20240502 14:06:44.742156 16089 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.11.9.67:37855: connect: Connection refused (error 111)
W20240502 14:06:44.742420 16089 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.11.9.67:37855: connect: Connection refused (error 111)
I20240502 14:06:44.843710 16437 heartbeater.cc:503] Master 127.11.9.126:34149 was elected leader, sending a full tablet report...
W20240502 14:06:45.780663 16089 auto_rebalancer.cc:249] could not retrieve auto-rebalancing replica moves: Not found: table d0b54d7b2eed411486693626af79f4c3: could not find any suitable replica to move from server 591c3b60a468458eb2f5ba7cfccdc087 to server 79a2dede1857429e96fe526b406a2b88
W20240502 14:06:46.791772 16089 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.11.9.67:37855: connect: Connection refused (error 111)
W20240502 14:06:46.794579 16025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.9.67:37855: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240502 14:06:46.798820 16089 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.11.9.67:37855: connect: Connection refused (error 111)
W20240502 14:06:46.799036 16089 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.11.9.67:37855: connect: Connection refused (error 111)
W20240502 14:06:46.802461 16089 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.11.9.65:43963: connect: Connection refused (error 111)
W20240502 14:06:46.802654 16089 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.11.9.65:43963: connect: Connection refused (error 111)
W20240502 14:06:46.805932 16089 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.11.9.67:37855: connect: Connection refused (error 111)
W20240502 14:06:46.806134 16089 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.11.9.67:37855: connect: Connection refused (error 111)
W20240502 14:06:47.817416 16089 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.11.9.67:37855: connect: Connection refused (error 111)
W20240502 14:06:47.822602 16089 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.11.9.67:37855: connect: Connection refused (error 111)
W20240502 14:06:47.822839 16089 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.11.9.67:37855: connect: Connection refused (error 111)
W20240502 14:06:47.826675 16089 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.11.9.65:43963: connect: Connection refused (error 111)
W20240502 14:06:47.826900 16089 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.11.9.65:43963: connect: Connection refused (error 111)
W20240502 14:06:47.830624 16089 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.11.9.67:37855: connect: Connection refused (error 111)
W20240502 14:06:47.830832 16089 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.11.9.67:37855: connect: Connection refused (error 111)
W20240502 14:06:48.832850 16089 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 4aef7125ff234f1e9f4ab6ed3afa2b30 not available for placement
W20240502 14:06:49.834781 16089 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 4aef7125ff234f1e9f4ab6ed3afa2b30 not available for placement
I20240502 14:06:49.862182 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:49.868515 16450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:49.869688 16451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:49.870640 16453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:49.874677 11301 server_base.cc:1034] running on GCE node
I20240502 14:06:49.877408 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:06:49.877650 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:49.877815 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658809877797 us; error 0 us; skew 500 ppm
I20240502 14:06:49.878332 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:49.885375 11301 webserver.cc:460] Webserver started at http://127.11.9.65:39033/ using document root <none> and password file <none>
I20240502 14:06:49.885862 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:49.886018 11301 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240502 14:06:49.889601 11301 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:06:49.892307 16458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:49.893064 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:06:49.893450 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-0-root
uuid: "825bd1431f0845bbbc6fe2b99bd4df13"
format_stamp: "Formatted at 2024-05-02 14:06:41 on dist-test-slave-j72s"
I20240502 14:06:49.893821 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:49.909292 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:49.910312 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:49.911659 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:49.915045 16465 ts_tablet_manager.cc:522] Loading tablet metadata (0/4 complete)
I20240502 14:06:49.933475 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240502 14:06:49.933771 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.020s user 0.000s sys 0.001s
I20240502 14:06:49.934006 11301 ts_tablet_manager.cc:574] Registering tablets (0/4 complete)
I20240502 14:06:49.938555 16465 tablet_bootstrap.cc:492] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13: Bootstrap starting.
I20240502 14:06:49.947649 16465 tablet_bootstrap.cc:492] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:49.947849 11301 ts_tablet_manager.cc:590] Registered 4 tablets
I20240502 14:06:49.948091 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.014s user 0.012s sys 0.000s
I20240502 14:06:49.948290 16465 tablet_bootstrap.cc:492] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13: Bootstrap complete.
I20240502 14:06:49.948835 16465 ts_tablet_manager.cc:1367] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240502 14:06:49.950549 16465 raft_consensus.cc:348] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:49.951072 16465 raft_consensus.cc:729] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 825bd1431f0845bbbc6fe2b99bd4df13, State: Initialized, Role: FOLLOWER
I20240502 14:06:49.951685 16465 consensus_queue.cc:260] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:49.953908 16465 ts_tablet_manager.cc:1398] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20240502 14:06:49.954600 16465 tablet_bootstrap.cc:492] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13: Bootstrap starting.
I20240502 14:06:49.966344 16465 tablet_bootstrap.cc:492] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:49.967296 16465 tablet_bootstrap.cc:492] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13: Bootstrap complete.
I20240502 14:06:49.968080 16465 ts_tablet_manager.cc:1367] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.001s
I20240502 14:06:49.970846 16465 raft_consensus.cc:348] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:49.971668 16465 raft_consensus.cc:729] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 825bd1431f0845bbbc6fe2b99bd4df13, State: Initialized, Role: FOLLOWER
I20240502 14:06:49.972620 16465 consensus_queue.cc:260] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:49.975688 16465 ts_tablet_manager.cc:1398] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13: Time spent starting tablet: real 0.007s user 0.006s sys 0.003s
I20240502 14:06:49.976845 16465 tablet_bootstrap.cc:492] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13: Bootstrap starting.
I20240502 14:06:49.992354 16465 tablet_bootstrap.cc:492] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:49.993284 16465 tablet_bootstrap.cc:492] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13: Bootstrap complete.
I20240502 14:06:49.994139 16465 ts_tablet_manager.cc:1367] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.016s
I20240502 14:06:49.997022 16465 raft_consensus.cc:348] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:49.997818 16465 raft_consensus.cc:729] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 825bd1431f0845bbbc6fe2b99bd4df13, State: Initialized, Role: FOLLOWER
I20240502 14:06:49.998692 16465 consensus_queue.cc:260] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:50.000921 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.65:43963
I20240502 14:06:50.001008 16525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.65:43963 every 8 connection(s)
I20240502 14:06:50.001216 16465 ts_tablet_manager.cc:1398] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13: Time spent starting tablet: real 0.007s user 0.000s sys 0.004s
I20240502 14:06:50.002099 16465 tablet_bootstrap.cc:492] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13: Bootstrap starting.
W20240502 14:06:50.005569 16527 tablet.cc:2369] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:06:50.006207 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:50.018949 16530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:50.020390 16532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:50.023485 16465 tablet_bootstrap.cc:492] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:06:50.024472 16465 tablet_bootstrap.cc:492] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13: Bootstrap complete.
I20240502 14:06:50.025180 16465 ts_tablet_manager.cc:1367] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.007s
I20240502 14:06:50.026986 16526 heartbeater.cc:347] Connected to a master server at 127.11.9.126:34149
I20240502 14:06:50.027606 11301 server_base.cc:1034] running on GCE node
I20240502 14:06:50.027671 16526 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:50.027853 16465 raft_consensus.cc:348] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:50.028707 16526 heartbeater.cc:511] Master 127.11.9.126:34149 requested a full tablet report, sending...
I20240502 14:06:50.028743 16465 raft_consensus.cc:729] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 825bd1431f0845bbbc6fe2b99bd4df13, State: Initialized, Role: FOLLOWER
W20240502 14:06:50.029165 16535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:50.029657 16465 consensus_queue.cc:260] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:50.030121 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:06:50.030391 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:50.030652 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658810030631 us; error 0 us; skew 500 ppm
I20240502 14:06:50.031244 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:50.031880 16465 ts_tablet_manager.cc:1398] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 14:06:50.034282 11301 webserver.cc:460] Webserver started at http://127.11.9.66:46113/ using document root <none> and password file <none>
I20240502 14:06:50.034727 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:50.034897 11301 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240502 14:06:50.034973 16038 ts_manager.cc:194] Re-registered known tserver with Master: 825bd1431f0845bbbc6fe2b99bd4df13 (127.11.9.65:43963)
I20240502 14:06:50.038491 11301 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20240502 14:06:50.039039 16038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37392
I20240502 14:06:50.041239 16526 heartbeater.cc:503] Master 127.11.9.126:34149 was elected leader, sending a full tablet report...
I20240502 14:06:50.041712 16539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:50.042564 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:06:50.042825 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-1-root
uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30"
format_stamp: "Formatted at 2024-05-02 14:06:41 on dist-test-slave-j72s"
I20240502 14:06:50.043083 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:50.086282 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:50.087314 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:50.088671 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:50.109987 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240502 14:06:50.110275 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.019s user 0.001s sys 0.000s
I20240502 14:06:50.114526 16546 tablet_bootstrap.cc:492] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30: Bootstrap starting.
I20240502 14:06:50.123756 16546 tablet_bootstrap.cc:492] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:50.124311 16546 tablet_bootstrap.cc:492] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30: Bootstrap complete.
I20240502 14:06:50.124321 11301 ts_tablet_manager.cc:590] Registered 4 tablets
I20240502 14:06:50.124684 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.014s user 0.014s sys 0.000s
I20240502 14:06:50.124909 16546 ts_tablet_manager.cc:1367] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20240502 14:06:50.126896 16546 raft_consensus.cc:348] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:50.127413 16546 raft_consensus.cc:729] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aef7125ff234f1e9f4ab6ed3afa2b30, State: Initialized, Role: FOLLOWER
I20240502 14:06:50.127995 16546 consensus_queue.cc:260] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:50.130846 16546 ts_tablet_manager.cc:1398] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20240502 14:06:50.131630 16546 tablet_bootstrap.cc:492] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30: Bootstrap starting.
I20240502 14:06:50.142077 16546 tablet_bootstrap.cc:492] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:50.142906 16546 tablet_bootstrap.cc:492] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30: Bootstrap complete.
I20240502 14:06:50.143709 16546 ts_tablet_manager.cc:1367] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.002s
I20240502 14:06:50.146772 16546 raft_consensus.cc:348] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:50.147662 16546 raft_consensus.cc:729] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aef7125ff234f1e9f4ab6ed3afa2b30, State: Initialized, Role: FOLLOWER
I20240502 14:06:50.148458 16546 consensus_queue.cc:260] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:50.150877 16546 ts_tablet_manager.cc:1398] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 14:06:50.151782 16546 tablet_bootstrap.cc:492] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30: Bootstrap starting.
I20240502 14:06:50.166561 16546 tablet_bootstrap.cc:492] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:50.167624 16546 tablet_bootstrap.cc:492] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30: Bootstrap complete.
I20240502 14:06:50.168430 16546 ts_tablet_manager.cc:1367] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20240502 14:06:50.171062 16546 raft_consensus.cc:348] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:50.171931 16546 raft_consensus.cc:729] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aef7125ff234f1e9f4ab6ed3afa2b30, State: Initialized, Role: FOLLOWER
I20240502 14:06:50.172603 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.66:42103
I20240502 14:06:50.172739 16606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.66:42103 every 8 connection(s)
I20240502 14:06:50.172811 16546 consensus_queue.cc:260] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:50.175076 16546 ts_tablet_manager.cc:1398] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240502 14:06:50.175927 16546 tablet_bootstrap.cc:492] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30: Bootstrap starting.
I20240502 14:06:50.177619 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:50.184389 16610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:50.190933 16607 heartbeater.cc:347] Connected to a master server at 127.11.9.126:34149
I20240502 14:06:50.191363 16607 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:50.192312 16607 heartbeater.cc:511] Master 127.11.9.126:34149 requested a full tablet report, sending...
W20240502 14:06:50.193332 16611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:50.194130 16546 tablet_bootstrap.cc:492] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240502 14:06:50.194173 16614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:50.195169 16546 tablet_bootstrap.cc:492] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30: Bootstrap complete.
I20240502 14:06:50.195832 16546 ts_tablet_manager.cc:1367] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.004s
I20240502 14:06:50.196156 11301 server_base.cc:1034] running on GCE node
I20240502 14:06:50.196715 16038 ts_manager.cc:194] Re-registered known tserver with Master: 4aef7125ff234f1e9f4ab6ed3afa2b30 (127.11.9.66:42103)
I20240502 14:06:50.197634 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:06:50.197930 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:50.198158 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658810198128 us; error 0 us; skew 500 ppm
I20240502 14:06:50.198149 16546 raft_consensus.cc:348] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:50.198880 16546 raft_consensus.cc:729] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aef7125ff234f1e9f4ab6ed3afa2b30, State: Initialized, Role: FOLLOWER
I20240502 14:06:50.199115 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:50.199462 16546 consensus_queue.cc:260] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:50.201328 16546 ts_tablet_manager.cc:1398] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:06:50.203775 16038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37408
I20240502 14:06:50.204731 11301 webserver.cc:460] Webserver started at http://127.11.9.67:46637/ using document root <none> and password file <none>
I20240502 14:06:50.205323 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:50.205482 11301 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240502 14:06:50.206298 16607 heartbeater.cc:503] Master 127.11.9.126:34149 was elected leader, sending a full tablet report...
I20240502 14:06:50.208858 11301 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20240502 14:06:50.212823 16619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:50.213672 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:06:50.214040 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-2-root
uuid: "591c3b60a468458eb2f5ba7cfccdc087"
format_stamp: "Formatted at 2024-05-02 14:06:41 on dist-test-slave-j72s"
I20240502 14:06:50.214377 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:50.231088 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:50.232000 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:50.233248 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:50.254595 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240502 14:06:50.254853 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.019s user 0.001s sys 0.000s
I20240502 14:06:50.258894 16626 tablet_bootstrap.cc:492] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087: Bootstrap starting.
I20240502 14:06:50.269150 11301 ts_tablet_manager.cc:590] Registered 4 tablets
I20240502 14:06:50.269371 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.014s user 0.011s sys 0.000s
I20240502 14:06:50.269618 16626 tablet_bootstrap.cc:492] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:06:50.270500 16626 tablet_bootstrap.cc:492] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087: Bootstrap complete.
I20240502 14:06:50.271049 16626 ts_tablet_manager.cc:1367] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20240502 14:06:50.272802 16626 raft_consensus.cc:348] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:50.273475 16626 raft_consensus.cc:729] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591c3b60a468458eb2f5ba7cfccdc087, State: Initialized, Role: FOLLOWER
I20240502 14:06:50.274078 16626 consensus_queue.cc:260] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:50.277089 16626 ts_tablet_manager.cc:1398] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20240502 14:06:50.277807 16626 tablet_bootstrap.cc:492] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087: Bootstrap starting.
I20240502 14:06:50.290547 16626 tablet_bootstrap.cc:492] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:06:50.291711 16626 tablet_bootstrap.cc:492] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087: Bootstrap complete.
I20240502 14:06:50.292469 16626 ts_tablet_manager.cc:1367] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20240502 14:06:50.295343 16626 raft_consensus.cc:348] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:50.296554 16626 raft_consensus.cc:729] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591c3b60a468458eb2f5ba7cfccdc087, State: Initialized, Role: FOLLOWER
I20240502 14:06:50.297477 16626 consensus_queue.cc:260] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:50.300137 16626 ts_tablet_manager.cc:1398] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20240502 14:06:50.301208 16626 tablet_bootstrap.cc:492] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087: Bootstrap starting.
I20240502 14:06:50.318513 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.67:37855
I20240502 14:06:50.318588 16686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.67:37855 every 8 connection(s)
I20240502 14:06:50.319386 16626 tablet_bootstrap.cc:492] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:06:50.320439 16626 tablet_bootstrap.cc:492] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087: Bootstrap complete.
I20240502 14:06:50.321125 16626 ts_tablet_manager.cc:1367] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.003s
I20240502 14:06:50.324126 16626 raft_consensus.cc:348] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:50.325452 16626 raft_consensus.cc:729] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591c3b60a468458eb2f5ba7cfccdc087, State: Initialized, Role: FOLLOWER
I20240502 14:06:50.326576 16626 consensus_queue.cc:260] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:50.328953 16626 ts_tablet_manager.cc:1398] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20240502 14:06:50.329797 16626 tablet_bootstrap.cc:492] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087: Bootstrap starting.
I20240502 14:06:50.334496 16687 heartbeater.cc:347] Connected to a master server at 127.11.9.126:34149
I20240502 14:06:50.334925 16687 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:50.335669 16687 heartbeater.cc:511] Master 127.11.9.126:34149 requested a full tablet report, sending...
I20240502 14:06:50.339084 16038 ts_manager.cc:194] Re-registered known tserver with Master: 591c3b60a468458eb2f5ba7cfccdc087 (127.11.9.67:37855)
I20240502 14:06:50.341233 16626 tablet_bootstrap.cc:492] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:50.342046 16626 tablet_bootstrap.cc:492] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087: Bootstrap complete.
I20240502 14:06:50.342932 16626 ts_tablet_manager.cc:1367] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240502 14:06:50.343663 16038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37418
I20240502 14:06:50.345232 16626 raft_consensus.cc:348] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:50.346024 16626 raft_consensus.cc:729] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591c3b60a468458eb2f5ba7cfccdc087, State: Initialized, Role: FOLLOWER
I20240502 14:06:50.346119 16687 heartbeater.cc:503] Master 127.11.9.126:34149 was elected leader, sending a full tablet report...
I20240502 14:06:50.347028 16626 consensus_queue.cc:260] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:50.348929 16626 ts_tablet_manager.cc:1398] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20240502 14:06:50.854413 16089 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 591c3b60a468458eb2f5ba7cfccdc087 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } } }
W20240502 14:06:50.857631 16089 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 0d10c138d696478daff9fbf0a7571c45, TS 825bd1431f0845bbbc6fe2b99bd4df13 -> TS 79a2dede1857429e96fe526b406a2b88 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
W20240502 14:06:50.857856 16089 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 0d10c138d696478daff9fbf0a7571c45, TS 825bd1431f0845bbbc6fe2b99bd4df13 -> TS 79a2dede1857429e96fe526b406a2b88 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
W20240502 14:06:50.868528 16089 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 473744e35b1a4da2ad3bac29156bcd9c, TS 591c3b60a468458eb2f5ba7cfccdc087 -> TS 79a2dede1857429e96fe526b406a2b88 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
W20240502 14:06:50.868724 16089 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 473744e35b1a4da2ad3bac29156bcd9c, TS 591c3b60a468458eb2f5ba7cfccdc087 -> TS 79a2dede1857429e96fe526b406a2b88 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
W20240502 14:06:50.871349 16089 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 9ce141f797a744a4a44c6ade10abe773, TS 4aef7125ff234f1e9f4ab6ed3afa2b30 -> TS 79a2dede1857429e96fe526b406a2b88 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
W20240502 14:06:50.871554 16089 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 9ce141f797a744a4a44c6ade10abe773, TS 4aef7125ff234f1e9f4ab6ed3afa2b30 -> TS 79a2dede1857429e96fe526b406a2b88 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:51.144320 16695 raft_consensus.cc:482] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:51.145274 16695 raft_consensus.cc:504] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:51.147756 16695 leader_election.cc:288] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4aef7125ff234f1e9f4ab6ed3afa2b30 (127.11.9.66:42103), 591c3b60a468458eb2f5ba7cfccdc087 (127.11.9.67:37855)
I20240502 14:06:51.161907 16661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2a87ab15cd7497abd4b275a559d7bba" candidate_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "591c3b60a468458eb2f5ba7cfccdc087" is_pre_election: true
I20240502 14:06:51.161965 16581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2a87ab15cd7497abd4b275a559d7bba" candidate_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" is_pre_election: true
I20240502 14:06:51.163306 16661 raft_consensus.cc:2396] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 825bd1431f0845bbbc6fe2b99bd4df13 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240502 14:06:51.163231 16581 raft_consensus.cc:2454] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 825bd1431f0845bbbc6fe2b99bd4df13 in term 1.
I20240502 14:06:51.164453 16462 leader_election.cc:302] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aef7125ff234f1e9f4ab6ed3afa2b30, 825bd1431f0845bbbc6fe2b99bd4df13; no voters:
I20240502 14:06:51.165138 16695 raft_consensus.cc:2781] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:06:51.165436 16695 raft_consensus.cc:482] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:51.165661 16695 raft_consensus.cc:3037] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:51.172202 16695 raft_consensus.cc:504] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:51.173583 16695 leader_election.cc:288] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [CANDIDATE]: Term 2 election: Requested vote from peers 4aef7125ff234f1e9f4ab6ed3afa2b30 (127.11.9.66:42103), 591c3b60a468458eb2f5ba7cfccdc087 (127.11.9.67:37855)
I20240502 14:06:51.174259 16581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2a87ab15cd7497abd4b275a559d7bba" candidate_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30"
I20240502 14:06:51.174576 16661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2a87ab15cd7497abd4b275a559d7bba" candidate_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "591c3b60a468458eb2f5ba7cfccdc087"
I20240502 14:06:51.174703 16581 raft_consensus.cc:3037] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:51.175063 16661 raft_consensus.cc:3037] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:51.180895 16581 raft_consensus.cc:2454] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 825bd1431f0845bbbc6fe2b99bd4df13 in term 2.
I20240502 14:06:51.181767 16661 raft_consensus.cc:2396] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 825bd1431f0845bbbc6fe2b99bd4df13 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240502 14:06:51.181798 16462 leader_election.cc:302] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aef7125ff234f1e9f4ab6ed3afa2b30, 825bd1431f0845bbbc6fe2b99bd4df13; no voters:
I20240502 14:06:51.182716 16695 raft_consensus.cc:2781] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:06:51.183745 16695 raft_consensus.cc:686] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [term 2 LEADER]: Becoming Leader. State: Replica: 825bd1431f0845bbbc6fe2b99bd4df13, State: Running, Role: LEADER
I20240502 14:06:51.184756 16695 consensus_queue.cc:237] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:51.192137 16038 catalog_manager.cc:5484] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 reported cstate change: term changed from 1 to 2, leader changed from 591c3b60a468458eb2f5ba7cfccdc087 (127.11.9.67) to 825bd1431f0845bbbc6fe2b99bd4df13 (127.11.9.65). New cstate: current_term: 2 leader_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:51.365458 16695 raft_consensus.cc:482] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:51.366004 16695 raft_consensus.cc:504] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:51.367830 16695 leader_election.cc:288] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4aef7125ff234f1e9f4ab6ed3afa2b30 (127.11.9.66:42103), 591c3b60a468458eb2f5ba7cfccdc087 (127.11.9.67:37855)
I20240502 14:06:51.368490 16581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d10c138d696478daff9fbf0a7571c45" candidate_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" is_pre_election: true
I20240502 14:06:51.368757 16661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d10c138d696478daff9fbf0a7571c45" candidate_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "591c3b60a468458eb2f5ba7cfccdc087" is_pre_election: true
I20240502 14:06:51.369046 16581 raft_consensus.cc:2454] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 825bd1431f0845bbbc6fe2b99bd4df13 in term 1.
I20240502 14:06:51.369666 16661 raft_consensus.cc:2396] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 825bd1431f0845bbbc6fe2b99bd4df13 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240502 14:06:51.370038 16462 leader_election.cc:302] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aef7125ff234f1e9f4ab6ed3afa2b30, 825bd1431f0845bbbc6fe2b99bd4df13; no voters:
I20240502 14:06:51.370666 16695 raft_consensus.cc:2781] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:06:51.370899 16695 raft_consensus.cc:482] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:51.371086 16695 raft_consensus.cc:3037] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:51.375649 16695 raft_consensus.cc:504] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:51.376838 16695 leader_election.cc:288] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [CANDIDATE]: Term 2 election: Requested vote from peers 4aef7125ff234f1e9f4ab6ed3afa2b30 (127.11.9.66:42103), 591c3b60a468458eb2f5ba7cfccdc087 (127.11.9.67:37855)
I20240502 14:06:51.377599 16581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d10c138d696478daff9fbf0a7571c45" candidate_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30"
I20240502 14:06:51.377729 16661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d10c138d696478daff9fbf0a7571c45" candidate_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "591c3b60a468458eb2f5ba7cfccdc087"
I20240502 14:06:51.378075 16581 raft_consensus.cc:3037] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:51.378240 16661 raft_consensus.cc:3037] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:51.382663 16581 raft_consensus.cc:2454] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 825bd1431f0845bbbc6fe2b99bd4df13 in term 2.
I20240502 14:06:51.382846 16661 raft_consensus.cc:2396] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 825bd1431f0845bbbc6fe2b99bd4df13 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240502 14:06:51.383489 16462 leader_election.cc:302] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aef7125ff234f1e9f4ab6ed3afa2b30, 825bd1431f0845bbbc6fe2b99bd4df13; no voters:
I20240502 14:06:51.384195 16695 raft_consensus.cc:2781] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:06:51.384562 16695 raft_consensus.cc:686] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 2 LEADER]: Becoming Leader. State: Replica: 825bd1431f0845bbbc6fe2b99bd4df13, State: Running, Role: LEADER
I20240502 14:06:51.385288 16695 consensus_queue.cc:237] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:51.390996 16038 catalog_manager.cc:5484] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 reported cstate change: term changed from 1 to 2, leader changed from 591c3b60a468458eb2f5ba7cfccdc087 (127.11.9.67) to 825bd1431f0845bbbc6fe2b99bd4df13 (127.11.9.65). New cstate: current_term: 2 leader_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:51.656692 16581 raft_consensus.cc:1261] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 2 FOLLOWER]: Refusing update from remote peer 825bd1431f0845bbbc6fe2b99bd4df13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 14:06:51.657979 16695 consensus_queue.cc:1035] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:51.668790 16661 raft_consensus.cc:1261] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 2 FOLLOWER]: Refusing update from remote peer 825bd1431f0845bbbc6fe2b99bd4df13: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20240502 14:06:51.669193 16661 pending_rounds.cc:76] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087: Aborting all ops after (but not including) 0
I20240502 14:06:51.669422 16661 pending_rounds.cc:97] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20240502 14:06:51.669711 16661 raft_consensus.cc:2866] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 14:06:51.671555 16695 consensus_queue.cc:1035] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:51.688903 16706 raft_consensus.cc:482] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:51.689363 16706 raft_consensus.cc:504] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:51.691340 16706 leader_election.cc:288] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 825bd1431f0845bbbc6fe2b99bd4df13 (127.11.9.65:43963), 591c3b60a468458eb2f5ba7cfccdc087 (127.11.9.67:37855)
I20240502 14:06:51.716428 16661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce141f797a744a4a44c6ade10abe773" candidate_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "591c3b60a468458eb2f5ba7cfccdc087" is_pre_election: true
I20240502 14:06:51.717020 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce141f797a744a4a44c6ade10abe773" candidate_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" is_pre_election: true
I20240502 14:06:51.717316 16661 raft_consensus.cc:2396] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4aef7125ff234f1e9f4ab6ed3afa2b30 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240502 14:06:51.717705 16500 raft_consensus.cc:2454] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aef7125ff234f1e9f4ab6ed3afa2b30 in term 1.
I20240502 14:06:51.718750 16543 leader_election.cc:302] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4aef7125ff234f1e9f4ab6ed3afa2b30, 825bd1431f0845bbbc6fe2b99bd4df13; no voters: 591c3b60a468458eb2f5ba7cfccdc087
I20240502 14:06:51.719432 16706 raft_consensus.cc:2781] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:06:51.719741 16706 raft_consensus.cc:482] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:51.719980 16706 raft_consensus.cc:3037] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:51.724666 16706 raft_consensus.cc:504] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:51.725973 16706 leader_election.cc:288] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [CANDIDATE]: Term 2 election: Requested vote from peers 825bd1431f0845bbbc6fe2b99bd4df13 (127.11.9.65:43963), 591c3b60a468458eb2f5ba7cfccdc087 (127.11.9.67:37855)
I20240502 14:06:51.726790 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce141f797a744a4a44c6ade10abe773" candidate_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "825bd1431f0845bbbc6fe2b99bd4df13"
I20240502 14:06:51.726881 16661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce141f797a744a4a44c6ade10abe773" candidate_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "591c3b60a468458eb2f5ba7cfccdc087"
I20240502 14:06:51.727274 16500 raft_consensus.cc:3037] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:51.727398 16661 raft_consensus.cc:3037] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:51.732394 16661 raft_consensus.cc:2396] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4aef7125ff234f1e9f4ab6ed3afa2b30 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240502 14:06:51.732359 16500 raft_consensus.cc:2454] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aef7125ff234f1e9f4ab6ed3afa2b30 in term 2.
I20240502 14:06:51.733337 16543 leader_election.cc:302] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aef7125ff234f1e9f4ab6ed3afa2b30, 825bd1431f0845bbbc6fe2b99bd4df13; no voters:
I20240502 14:06:51.733989 16706 raft_consensus.cc:2781] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:06:51.735054 16706 raft_consensus.cc:686] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 2 LEADER]: Becoming Leader. State: Replica: 4aef7125ff234f1e9f4ab6ed3afa2b30, State: Running, Role: LEADER
I20240502 14:06:51.735855 16706 consensus_queue.cc:237] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:51.742267 16039 catalog_manager.cc:5484] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 reported cstate change: term changed from 1 to 2, leader changed from 591c3b60a468458eb2f5ba7cfccdc087 (127.11.9.67) to 4aef7125ff234f1e9f4ab6ed3afa2b30 (127.11.9.66). New cstate: current_term: 2 leader_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:51.781350 16695 raft_consensus.cc:482] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:51.781936 16695 raft_consensus.cc:504] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:51.783618 16695 leader_election.cc:288] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4aef7125ff234f1e9f4ab6ed3afa2b30 (127.11.9.66:42103), 591c3b60a468458eb2f5ba7cfccdc087 (127.11.9.67:37855)
I20240502 14:06:51.784351 16581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "473744e35b1a4da2ad3bac29156bcd9c" candidate_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" is_pre_election: true
I20240502 14:06:51.785013 16581 raft_consensus.cc:2454] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 825bd1431f0845bbbc6fe2b99bd4df13 in term 1.
I20240502 14:06:51.784940 16661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "473744e35b1a4da2ad3bac29156bcd9c" candidate_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "591c3b60a468458eb2f5ba7cfccdc087" is_pre_election: true
I20240502 14:06:51.785871 16661 raft_consensus.cc:2454] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 825bd1431f0845bbbc6fe2b99bd4df13 in term 1.
I20240502 14:06:51.786161 16462 leader_election.cc:302] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aef7125ff234f1e9f4ab6ed3afa2b30, 825bd1431f0845bbbc6fe2b99bd4df13; no voters:
I20240502 14:06:51.786911 16695 raft_consensus.cc:2781] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:06:51.787180 16695 raft_consensus.cc:482] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:51.787433 16695 raft_consensus.cc:3037] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:51.791867 16695 raft_consensus.cc:504] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:51.793397 16695 leader_election.cc:288] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [CANDIDATE]: Term 2 election: Requested vote from peers 4aef7125ff234f1e9f4ab6ed3afa2b30 (127.11.9.66:42103), 591c3b60a468458eb2f5ba7cfccdc087 (127.11.9.67:37855)
I20240502 14:06:51.794036 16581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "473744e35b1a4da2ad3bac29156bcd9c" candidate_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30"
I20240502 14:06:51.794266 16661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "473744e35b1a4da2ad3bac29156bcd9c" candidate_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "591c3b60a468458eb2f5ba7cfccdc087"
I20240502 14:06:51.794481 16581 raft_consensus.cc:3037] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:51.794801 16661 raft_consensus.cc:3037] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:51.798872 16581 raft_consensus.cc:2454] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 825bd1431f0845bbbc6fe2b99bd4df13 in term 2.
I20240502 14:06:51.799204 16661 raft_consensus.cc:2454] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 825bd1431f0845bbbc6fe2b99bd4df13 in term 2.
I20240502 14:06:51.799757 16462 leader_election.cc:302] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aef7125ff234f1e9f4ab6ed3afa2b30, 825bd1431f0845bbbc6fe2b99bd4df13; no voters:
I20240502 14:06:51.800514 16695 raft_consensus.cc:2781] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:06:51.800848 16695 raft_consensus.cc:686] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [term 2 LEADER]: Becoming Leader. State: Replica: 825bd1431f0845bbbc6fe2b99bd4df13, State: Running, Role: LEADER
I20240502 14:06:51.801476 16695 consensus_queue.cc:237] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:51.807106 16039 catalog_manager.cc:5484] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:51.881928 16500 consensus_queue.cc:237] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } attrs { replace: true } } peers { permanent_uuid: "79a2dede1857429e96fe526b406a2b88" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39969 } attrs { promote: true } }
I20240502 14:06:51.887774 16581 raft_consensus.cc:1261] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 2 FOLLOWER]: Refusing update from remote peer 825bd1431f0845bbbc6fe2b99bd4df13: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:06:51.890425 16661 raft_consensus.cc:1261] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 2 FOLLOWER]: Refusing update from remote peer 825bd1431f0845bbbc6fe2b99bd4df13: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:06:51.890745 16720 consensus_queue.cc:1035] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 14:06:51.891252 16660 raft_consensus.cc:1261] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [term 2 FOLLOWER]: Refusing update from remote peer 825bd1431f0845bbbc6fe2b99bd4df13: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20240502 14:06:51.891602 16660 pending_rounds.cc:76] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087: Aborting all ops after (but not including) 0
I20240502 14:06:51.891810 16660 pending_rounds.cc:97] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20240502 14:06:51.892062 16660 raft_consensus.cc:2866] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 14:06:51.892071 16695 consensus_queue.cc:1035] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 14:06:51.893687 16720 consensus_queue.cc:1035] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:51.898193 16695 raft_consensus.cc:2932] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 79a2dede1857429e96fe526b406a2b88 (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } attrs { replace: true } } peers { permanent_uuid: "79a2dede1857429e96fe526b406a2b88" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39969 } attrs { promote: true } } }
I20240502 14:06:51.899911 16581 raft_consensus.cc:2932] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 79a2dede1857429e96fe526b406a2b88 (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } attrs { replace: true } } peers { permanent_uuid: "79a2dede1857429e96fe526b406a2b88" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39969 } attrs { promote: true } } }
W20240502 14:06:51.902025 16459 consensus_peers.cc:485] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 -> Peer 79a2dede1857429e96fe526b406a2b88 (127.11.9.68:39969): Couldn't send request to peer 79a2dede1857429e96fe526b406a2b88. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e2a87ab15cd7497abd4b275a559d7bba. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:51.902524 16660 raft_consensus.cc:2932] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 79a2dede1857429e96fe526b406a2b88 (127.11.9.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } attrs { replace: true } } peers { permanent_uuid: "79a2dede1857429e96fe526b406a2b88" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39969 } attrs { promote: true } } }
I20240502 14:06:51.912518 16580 raft_consensus.cc:1261] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 2 FOLLOWER]: Refusing update from remote peer 825bd1431f0845bbbc6fe2b99bd4df13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 14:06:51.914139 16720 consensus_queue.cc:1035] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:06:51.916483 16089 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20240502 14:06:51.920183 16039 catalog_manager.cc:5484] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 reported cstate change: config changed from index -1 to 2, NON_VOTER 79a2dede1857429e96fe526b406a2b88 (127.11.9.68) added. New cstate: current_term: 2 leader_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79a2dede1857429e96fe526b406a2b88" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39969 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:06:51.927644 16089 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 473744e35b1a4da2ad3bac29156bcd9c, TS 4aef7125ff234f1e9f4ab6ed3afa2b30 -> TS 79a2dede1857429e96fe526b406a2b88 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
W20240502 14:06:51.928020 16089 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 473744e35b1a4da2ad3bac29156bcd9c, TS 4aef7125ff234f1e9f4ab6ed3afa2b30 -> TS 79a2dede1857429e96fe526b406a2b88 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
W20240502 14:06:51.952932 16089 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 9ce141f797a744a4a44c6ade10abe773, TS 825bd1431f0845bbbc6fe2b99bd4df13 -> TS 79a2dede1857429e96fe526b406a2b88 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
W20240502 14:06:51.953167 16089 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 9ce141f797a744a4a44c6ade10abe773, TS 825bd1431f0845bbbc6fe2b99bd4df13 -> TS 79a2dede1857429e96fe526b406a2b88 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:52.036514 16710 raft_consensus.cc:482] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:52.036991 16710 raft_consensus.cc:504] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:52.039404 16710 leader_election.cc:288] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4aef7125ff234f1e9f4ab6ed3afa2b30 (127.11.9.66:42103), 825bd1431f0845bbbc6fe2b99bd4df13 (127.11.9.65:43963)
I20240502 14:06:52.050999 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce141f797a744a4a44c6ade10abe773" candidate_uuid: "591c3b60a468458eb2f5ba7cfccdc087" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" is_pre_election: true
I20240502 14:06:52.051870 16500 raft_consensus.cc:2454] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 591c3b60a468458eb2f5ba7cfccdc087 in term 2.
I20240502 14:06:52.052130 16580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce141f797a744a4a44c6ade10abe773" candidate_uuid: "591c3b60a468458eb2f5ba7cfccdc087" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" is_pre_election: true
I20240502 14:06:52.052851 16623 leader_election.cc:302] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 591c3b60a468458eb2f5ba7cfccdc087, 825bd1431f0845bbbc6fe2b99bd4df13; no voters:
I20240502 14:06:52.053651 16710 raft_consensus.cc:2781] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 14:06:52.053984 16710 raft_consensus.cc:482] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:52.054286 16710 raft_consensus.cc:3037] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:06:52.060025 16710 raft_consensus.cc:504] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:52.061457 16710 leader_election.cc:288] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [CANDIDATE]: Term 3 election: Requested vote from peers 4aef7125ff234f1e9f4ab6ed3afa2b30 (127.11.9.66:42103), 825bd1431f0845bbbc6fe2b99bd4df13 (127.11.9.65:43963)
I20240502 14:06:52.062089 16580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce141f797a744a4a44c6ade10abe773" candidate_uuid: "591c3b60a468458eb2f5ba7cfccdc087" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30"
I20240502 14:06:52.062266 16500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce141f797a744a4a44c6ade10abe773" candidate_uuid: "591c3b60a468458eb2f5ba7cfccdc087" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "825bd1431f0845bbbc6fe2b99bd4df13"
I20240502 14:06:52.062896 16500 raft_consensus.cc:3037] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:06:52.067692 16500 raft_consensus.cc:2454] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 591c3b60a468458eb2f5ba7cfccdc087 in term 3.
I20240502 14:06:52.068609 16623 leader_election.cc:302] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 591c3b60a468458eb2f5ba7cfccdc087, 825bd1431f0845bbbc6fe2b99bd4df13; no voters: 4aef7125ff234f1e9f4ab6ed3afa2b30
I20240502 14:06:52.069267 16710 raft_consensus.cc:2781] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:06:52.070317 16710 raft_consensus.cc:686] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 3 LEADER]: Becoming Leader. State: Replica: 591c3b60a468458eb2f5ba7cfccdc087, State: Running, Role: LEADER
I20240502 14:06:52.071034 16710 consensus_queue.cc:237] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:52.078233 16037 catalog_manager.cc:5484] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 reported cstate change: term changed from 2 to 3, leader changed from 4aef7125ff234f1e9f4ab6ed3afa2b30 (127.11.9.66) to 591c3b60a468458eb2f5ba7cfccdc087 (127.11.9.67). New cstate: current_term: 3 leader_uuid: "591c3b60a468458eb2f5ba7cfccdc087" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } health_report { overall_health: HEALTHY } } }
I20240502 14:06:52.183872 16660 raft_consensus.cc:1261] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087 [term 2 FOLLOWER]: Refusing update from remote peer 825bd1431f0845bbbc6fe2b99bd4df13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:06:52.185180 16724 consensus_queue.cc:1035] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:52.198354 16580 raft_consensus.cc:1261] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 2 FOLLOWER]: Refusing update from remote peer 825bd1431f0845bbbc6fe2b99bd4df13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:06:52.199998 16720 consensus_queue.cc:1035] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:06:52.270774 16660 raft_consensus.cc:1226] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 3 LEADER]: Rejecting Update request from peer 4aef7125ff234f1e9f4ab6ed3afa2b30 for earlier term 2. Current term is 3. Ops: []
I20240502 14:06:52.272074 16706 consensus_queue.cc:1046] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:52.273203 16706 consensus_queue.cc:1225] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [LEADER]: Peer 591c3b60a468458eb2f5ba7cfccdc087 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240502 14:06:52.273830 16734 raft_consensus.cc:3032] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:06:52.274142 16734 raft_consensus.cc:729] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4aef7125ff234f1e9f4ab6ed3afa2b30, State: Running, Role: LEADER
I20240502 14:06:52.274857 16734 consensus_queue.cc:260] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } }
I20240502 14:06:52.275970 16734 raft_consensus.cc:3037] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:06:52.306329 16735 ts_tablet_manager.cc:902] T e2a87ab15cd7497abd4b275a559d7bba P 79a2dede1857429e96fe526b406a2b88: Initiating tablet copy from peer 825bd1431f0845bbbc6fe2b99bd4df13 (127.11.9.65:43963)
I20240502 14:06:52.310343 16735 tablet_copy_client.cc:308] T e2a87ab15cd7497abd4b275a559d7bba P 79a2dede1857429e96fe526b406a2b88: tablet copy: Beginning tablet copy session from remote peer at address 127.11.9.65:43963
I20240502 14:06:52.320200 16510 tablet_copy_service.cc:140] P 825bd1431f0845bbbc6fe2b99bd4df13: Received BeginTabletCopySession request for tablet e2a87ab15cd7497abd4b275a559d7bba from peer 79a2dede1857429e96fe526b406a2b88 ({username='slave'} at 127.0.0.1:46538)
I20240502 14:06:52.320748 16510 tablet_copy_service.cc:161] P 825bd1431f0845bbbc6fe2b99bd4df13: Beginning new tablet copy session on tablet e2a87ab15cd7497abd4b275a559d7bba from peer 79a2dede1857429e96fe526b406a2b88 at {username='slave'} at 127.0.0.1:46538: session id = 79a2dede1857429e96fe526b406a2b88-e2a87ab15cd7497abd4b275a559d7bba
I20240502 14:06:52.327265 16510 tablet_copy_source_session.cc:205] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:06:52.329934 16735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2a87ab15cd7497abd4b275a559d7bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:52.342278 16735 tablet_copy_client.cc:783] T e2a87ab15cd7497abd4b275a559d7bba P 79a2dede1857429e96fe526b406a2b88: tablet copy: Starting download of 0 data blocks...
I20240502 14:06:52.342854 16735 tablet_copy_client.cc:647] T e2a87ab15cd7497abd4b275a559d7bba P 79a2dede1857429e96fe526b406a2b88: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:52.346201 16735 tablet_copy_client.cc:523] T e2a87ab15cd7497abd4b275a559d7bba P 79a2dede1857429e96fe526b406a2b88: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:52.354352 16735 tablet_bootstrap.cc:492] T e2a87ab15cd7497abd4b275a559d7bba P 79a2dede1857429e96fe526b406a2b88: Bootstrap starting.
I20240502 14:06:52.372308 16735 tablet_bootstrap.cc:492] T e2a87ab15cd7497abd4b275a559d7bba P 79a2dede1857429e96fe526b406a2b88: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:52.373211 16735 tablet_bootstrap.cc:492] T e2a87ab15cd7497abd4b275a559d7bba P 79a2dede1857429e96fe526b406a2b88: Bootstrap complete.
I20240502 14:06:52.373868 16735 ts_tablet_manager.cc:1367] T e2a87ab15cd7497abd4b275a559d7bba P 79a2dede1857429e96fe526b406a2b88: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.007s
I20240502 14:06:52.376401 16735 raft_consensus.cc:348] T e2a87ab15cd7497abd4b275a559d7bba P 79a2dede1857429e96fe526b406a2b88 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } attrs { replace: true } } peers { permanent_uuid: "79a2dede1857429e96fe526b406a2b88" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39969 } attrs { promote: true } }
I20240502 14:06:52.377180 16735 raft_consensus.cc:729] T e2a87ab15cd7497abd4b275a559d7bba P 79a2dede1857429e96fe526b406a2b88 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 79a2dede1857429e96fe526b406a2b88, State: Initialized, Role: LEARNER
I20240502 14:06:52.377759 16735 consensus_queue.cc:260] T e2a87ab15cd7497abd4b275a559d7bba P 79a2dede1857429e96fe526b406a2b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } attrs { replace: true } } peers { permanent_uuid: "79a2dede1857429e96fe526b406a2b88" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 39969 } attrs { promote: true } }
I20240502 14:06:52.380805 16735 ts_tablet_manager.cc:1398] T e2a87ab15cd7497abd4b275a559d7bba P 79a2dede1857429e96fe526b406a2b88: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20240502 14:06:52.382462 16510 tablet_copy_service.cc:342] P 825bd1431f0845bbbc6fe2b99bd4df13: Request end of tablet copy session 79a2dede1857429e96fe526b406a2b88-e2a87ab15cd7497abd4b275a559d7bba received from {username='slave'} at 127.0.0.1:46538
I20240502 14:06:52.382781 16510 tablet_copy_service.cc:433] P 825bd1431f0845bbbc6fe2b99bd4df13: ending tablet copy session 79a2dede1857429e96fe526b406a2b88-e2a87ab15cd7497abd4b275a559d7bba on tablet e2a87ab15cd7497abd4b275a559d7bba with peer 79a2dede1857429e96fe526b406a2b88
I20240502 14:06:52.479595 16499 raft_consensus.cc:1261] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 3 FOLLOWER]: Refusing update from remote peer 591c3b60a468458eb2f5ba7cfccdc087: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 14:06:52.480749 16710 consensus_queue.cc:1035] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [LEADER]: Connected to new peer: Peer: permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:52.492425 16580 raft_consensus.cc:1261] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 3 FOLLOWER]: Refusing update from remote peer 591c3b60a468458eb2f5ba7cfccdc087: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 14:06:52.493783 16710 consensus_queue.cc:1035] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:52.494518 16710 consensus_queue.cc:1225] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [LEADER]: Peer 4aef7125ff234f1e9f4ab6ed3afa2b30 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240502 14:06:52.497125 16580 pending_rounds.cc:76] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30: Aborting all ops after (but not including) 0
I20240502 14:06:52.497416 16580 pending_rounds.cc:97] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20240502 14:06:52.497638 16580 raft_consensus.cc:2866] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 14:06:52.794107 16412 raft_consensus.cc:1203] T e2a87ab15cd7497abd4b275a559d7bba P 79a2dede1857429e96fe526b406a2b88 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20240502 14:06:53.187441 16695 raft_consensus.cc:1050] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13: attempting to promote NON_VOTER 79a2dede1857429e96fe526b406a2b88 to VOTER
I20240502 14:06:53.189692 16695 consensus_queue.cc:237] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } attrs { replace: true } } peers { permanent_uuid: "79a2dede1857429e96fe526b406a2b88" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 39969 } attrs { promote: false } }
I20240502 14:06:53.195012 16412 raft_consensus.cc:1261] T e2a87ab15cd7497abd4b275a559d7bba P 79a2dede1857429e96fe526b406a2b88 [term 2 LEARNER]: Refusing update from remote peer 825bd1431f0845bbbc6fe2b99bd4df13: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:06:53.195186 16660 raft_consensus.cc:1261] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 2 FOLLOWER]: Refusing update from remote peer 825bd1431f0845bbbc6fe2b99bd4df13: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:06:53.195825 16580 raft_consensus.cc:1261] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 2 FOLLOWER]: Refusing update from remote peer 825bd1431f0845bbbc6fe2b99bd4df13: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:06:53.197244 16720 consensus_queue.cc:1035] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:53.198097 16724 consensus_queue.cc:1035] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:53.201660 16695 consensus_queue.cc:1035] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79a2dede1857429e96fe526b406a2b88" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 39969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:06:53.209718 16720 raft_consensus.cc:2932] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 79a2dede1857429e96fe526b406a2b88 (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } attrs { replace: true } } peers { permanent_uuid: "79a2dede1857429e96fe526b406a2b88" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 39969 } attrs { promote: false } } }
I20240502 14:06:53.211412 16580 raft_consensus.cc:2932] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 79a2dede1857429e96fe526b406a2b88 (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } attrs { replace: true } } peers { permanent_uuid: "79a2dede1857429e96fe526b406a2b88" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 39969 } attrs { promote: false } } }
I20240502 14:06:53.213793 16412 raft_consensus.cc:2932] T e2a87ab15cd7497abd4b275a559d7bba P 79a2dede1857429e96fe526b406a2b88 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 79a2dede1857429e96fe526b406a2b88 (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } attrs { replace: true } } peers { permanent_uuid: "79a2dede1857429e96fe526b406a2b88" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 39969 } attrs { promote: false } } }
I20240502 14:06:53.221470 16660 raft_consensus.cc:2932] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 79a2dede1857429e96fe526b406a2b88 (127.11.9.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } attrs { replace: true } } peers { permanent_uuid: "79a2dede1857429e96fe526b406a2b88" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 39969 } attrs { promote: false } } }
I20240502 14:06:53.224056 16037 catalog_manager.cc:5484] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 reported cstate change: config changed from index 2 to 3, 79a2dede1857429e96fe526b406a2b88 (127.11.9.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "591c3b60a468458eb2f5ba7cfccdc087" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 37855 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79a2dede1857429e96fe526b406a2b88" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 39969 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:06:53.245374 16499 consensus_queue.cc:237] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "79a2dede1857429e96fe526b406a2b88" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 39969 } attrs { promote: false } }
I20240502 14:06:53.249980 16412 raft_consensus.cc:1261] T e2a87ab15cd7497abd4b275a559d7bba P 79a2dede1857429e96fe526b406a2b88 [term 2 FOLLOWER]: Refusing update from remote peer 825bd1431f0845bbbc6fe2b99bd4df13: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:06:53.250520 16580 raft_consensus.cc:1261] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 2 FOLLOWER]: Refusing update from remote peer 825bd1431f0845bbbc6fe2b99bd4df13: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:06:53.251092 16749 consensus_queue.cc:1035] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79a2dede1857429e96fe526b406a2b88" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 39969 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:53.252398 16695 consensus_queue.cc:1035] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:06:53.256673 16749 raft_consensus.cc:2932] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 591c3b60a468458eb2f5ba7cfccdc087 (127.11.9.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "79a2dede1857429e96fe526b406a2b88" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 39969 } attrs { promote: false } } }
I20240502 14:06:53.260221 16412 raft_consensus.cc:2932] T e2a87ab15cd7497abd4b275a559d7bba P 79a2dede1857429e96fe526b406a2b88 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 591c3b60a468458eb2f5ba7cfccdc087 (127.11.9.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "79a2dede1857429e96fe526b406a2b88" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 39969 } attrs { promote: false } } }
I20240502 14:06:53.261559 16580 raft_consensus.cc:2932] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 591c3b60a468458eb2f5ba7cfccdc087 (127.11.9.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } } peers { permanent_uuid: "79a2dede1857429e96fe526b406a2b88" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 39969 } attrs { promote: false } } }
I20240502 14:06:53.265650 16088 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e2a87ab15cd7497abd4b275a559d7bba with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:06:53.271459 16041 catalog_manager.cc:5484] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 reported cstate change: config changed from index 3 to 4, VOTER 591c3b60a468458eb2f5ba7cfccdc087 (127.11.9.67) evicted. New cstate: current_term: 2 leader_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "825bd1431f0845bbbc6fe2b99bd4df13" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 43963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aef7125ff234f1e9f4ab6ed3afa2b30" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 42103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79a2dede1857429e96fe526b406a2b88" member_type: VOTER last_known_addr { host: "127.11.9.68" port: 39969 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:06:53.293473 16651 tablet_service.cc:1514] Processing DeleteTablet for tablet e2a87ab15cd7497abd4b275a559d7bba with delete_type TABLET_DATA_TOMBSTONED (TS 591c3b60a468458eb2f5ba7cfccdc087 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42898
I20240502 14:06:53.295331 16760 tablet_replica.cc:330] stopping tablet replica
I20240502 14:06:53.296018 16760 raft_consensus.cc:2229] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:53.296562 16760 raft_consensus.cc:2258] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:53.299002 16760 ts_tablet_manager.cc:1875] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:06:53.308696 16760 ts_tablet_manager.cc:1888] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 14:06:53.309065 16760 log.cc:1196] T e2a87ab15cd7497abd4b275a559d7bba P 591c3b60a468458eb2f5ba7cfccdc087: Deleting WAL directory at /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714658659912502-11301-0/minicluster-data/ts-2-root/wals/e2a87ab15cd7497abd4b275a559d7bba
I20240502 14:06:53.310413 16025 catalog_manager.cc:4830] TS 591c3b60a468458eb2f5ba7cfccdc087 (127.11.9.67:37855): tablet e2a87ab15cd7497abd4b275a559d7bba (table test-workload [id=d0b54d7b2eed411486693626af79f4c3]) successfully deleted
I20240502 14:06:53.842651 11301 tablet_server.cc:178] TabletServer@127.11.9.65:43963 shutting down...
I20240502 14:06:53.863247 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:06:53.863868 11301 tablet_replica.cc:330] stopping tablet replica
I20240502 14:06:53.864434 11301 raft_consensus.cc:2229] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [term 2 LEADER]: Raft consensus shutting down.
I20240502 14:06:53.865139 11301 raft_consensus.cc:2258] T 473744e35b1a4da2ad3bac29156bcd9c P 825bd1431f0845bbbc6fe2b99bd4df13 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:53.867041 11301 tablet_replica.cc:330] stopping tablet replica
I20240502 14:06:53.867524 11301 raft_consensus.cc:2229] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 2 LEADER]: Raft consensus shutting down.
I20240502 14:06:53.868201 11301 raft_consensus.cc:2258] T 0d10c138d696478daff9fbf0a7571c45 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:53.869925 11301 tablet_replica.cc:330] stopping tablet replica
I20240502 14:06:53.870350 11301 raft_consensus.cc:2229] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:53.870720 11301 raft_consensus.cc:2258] T 9ce141f797a744a4a44c6ade10abe773 P 825bd1431f0845bbbc6fe2b99bd4df13 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:53.872314 11301 tablet_replica.cc:330] stopping tablet replica
I20240502 14:06:53.872748 11301 raft_consensus.cc:2229] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [term 2 LEADER]: Raft consensus shutting down.
I20240502 14:06:53.873394 11301 raft_consensus.cc:2258] T e2a87ab15cd7497abd4b275a559d7bba P 825bd1431f0845bbbc6fe2b99bd4df13 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:53.895697 11301 tablet_server.cc:194] TabletServer@127.11.9.65:43963 shutdown complete.
I20240502 14:06:53.908111 11301 tablet_server.cc:178] TabletServer@127.11.9.66:42103 shutting down...
I20240502 14:06:53.929034 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:06:53.929708 11301 tablet_replica.cc:330] stopping tablet replica
I20240502 14:06:53.930272 11301 raft_consensus.cc:2229] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:53.930658 11301 raft_consensus.cc:2258] T 473744e35b1a4da2ad3bac29156bcd9c P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:53.932370 11301 tablet_replica.cc:330] stopping tablet replica
I20240502 14:06:53.932823 11301 raft_consensus.cc:2229] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:53.933182 11301 raft_consensus.cc:2258] T 0d10c138d696478daff9fbf0a7571c45 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:53.934952 11301 tablet_replica.cc:330] stopping tablet replica
I20240502 14:06:53.935395 11301 raft_consensus.cc:2229] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:53.935745 11301 raft_consensus.cc:2258] T 9ce141f797a744a4a44c6ade10abe773 P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:53.937291 11301 tablet_replica.cc:330] stopping tablet replica
I20240502 14:06:53.937727 11301 raft_consensus.cc:2229] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:53.938185 11301 raft_consensus.cc:2258] T e2a87ab15cd7497abd4b275a559d7bba P 4aef7125ff234f1e9f4ab6ed3afa2b30 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:53.959663 11301 tablet_server.cc:194] TabletServer@127.11.9.66:42103 shutdown complete.
I20240502 14:06:53.972007 11301 tablet_server.cc:178] TabletServer@127.11.9.67:37855 shutting down...
I20240502 14:06:53.998991 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:06:53.999626 11301 tablet_replica.cc:330] stopping tablet replica
I20240502 14:06:54.000114 11301 raft_consensus.cc:2229] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:54.000509 11301 raft_consensus.cc:2258] T 473744e35b1a4da2ad3bac29156bcd9c P 591c3b60a468458eb2f5ba7cfccdc087 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:54.002293 11301 tablet_replica.cc:330] stopping tablet replica
I20240502 14:06:54.002707 11301 raft_consensus.cc:2229] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:54.003053 11301 raft_consensus.cc:2258] T 0d10c138d696478daff9fbf0a7571c45 P 591c3b60a468458eb2f5ba7cfccdc087 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:54.004801 11301 tablet_replica.cc:330] stopping tablet replica
I20240502 14:06:54.005206 11301 raft_consensus.cc:2229] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 3 LEADER]: Raft consensus shutting down.
I20240502 14:06:54.006196 11301 raft_consensus.cc:2258] T 9ce141f797a744a4a44c6ade10abe773 P 591c3b60a468458eb2f5ba7cfccdc087 [term 3 FOLLOWER]: Raft consensus is shut down!
W20240502 14:06:54.008184 16623 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.9.65:43963: connect: Connection refused (error 111) [suppressed 5 similar messages]
I20240502 14:06:54.028445 11301 tablet_server.cc:194] TabletServer@127.11.9.67:37855 shutdown complete.
I20240502 14:06:54.041204 11301 tablet_server.cc:178] TabletServer@127.11.9.68:0 shutting down...
I20240502 14:06:54.059173 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:06:54.059823 11301 tablet_replica.cc:330] T e2a87ab15cd7497abd4b275a559d7bba P 79a2dede1857429e96fe526b406a2b88: stopping tablet replica
I20240502 14:06:54.060258 11301 raft_consensus.cc:2229] T e2a87ab15cd7497abd4b275a559d7bba P 79a2dede1857429e96fe526b406a2b88 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:54.060765 11301 raft_consensus.cc:2258] T e2a87ab15cd7497abd4b275a559d7bba P 79a2dede1857429e96fe526b406a2b88 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:54.079977 11301 tablet_server.cc:194] TabletServer@127.11.9.68:0 shutdown complete.
I20240502 14:06:54.088469 11301 master.cc:537] Master@127.11.9.126:34149 shutting down...
I20240502 14:06:54.104635 11301 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 16a279f430ec4b5297e6c162adef8731 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:54.105062 11301 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 16a279f430ec4b5297e6c162adef8731 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:54.105334 11301 tablet_replica.cc:330] T 00000000000000000000000000000000 P 16a279f430ec4b5297e6c162adef8731: stopping tablet replica
I20240502 14:06:54.124049 11301 master.cc:552] Master@127.11.9.126:34149 shutdown complete.
[ OK ] AutoRebalancerTest.TestHandlingFailedTservers (12572 ms)
[ RUN ] AutoRebalancerTest.TestDeletedTables
I20240502 14:06:54.157482 11301 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.9.126:36895
I20240502 14:06:54.158512 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:54.163785 16763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:54.164510 16764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:54.165096 16766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:54.166010 11301 server_base.cc:1034] running on GCE node
I20240502 14:06:54.167985 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:06:54.168166 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:54.168344 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658814168324 us; error 0 us; skew 500 ppm
I20240502 14:06:54.168830 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:54.171164 11301 webserver.cc:460] Webserver started at http://127.11.9.126:33317/ using document root <none> and password file <none>
I20240502 14:06:54.171594 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:54.171753 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:54.172004 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:54.173058 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/master-0-root/instance:
uuid: "854848cec8144d36b0e06dd804df9293"
format_stamp: "Formatted at 2024-05-02 14:06:54 on dist-test-slave-j72s"
I20240502 14:06:54.177196 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20240502 14:06:54.180291 16771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:54.180949 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:06:54.181187 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/master-0-root
uuid: "854848cec8144d36b0e06dd804df9293"
format_stamp: "Formatted at 2024-05-02 14:06:54 on dist-test-slave-j72s"
I20240502 14:06:54.181421 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:54.192874 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:54.193974 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:54.228024 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.126:36895
I20240502 14:06:54.228132 16822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.126:36895 every 8 connection(s)
I20240502 14:06:54.231297 16823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:54.240183 16823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 854848cec8144d36b0e06dd804df9293: Bootstrap starting.
I20240502 14:06:54.243666 16823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 854848cec8144d36b0e06dd804df9293: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:54.247256 16823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 854848cec8144d36b0e06dd804df9293: No bootstrap required, opened a new log
I20240502 14:06:54.249099 16823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 854848cec8144d36b0e06dd804df9293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "854848cec8144d36b0e06dd804df9293" member_type: VOTER }
I20240502 14:06:54.249454 16823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 854848cec8144d36b0e06dd804df9293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:54.249725 16823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 854848cec8144d36b0e06dd804df9293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854848cec8144d36b0e06dd804df9293, State: Initialized, Role: FOLLOWER
I20240502 14:06:54.250252 16823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 854848cec8144d36b0e06dd804df9293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "854848cec8144d36b0e06dd804df9293" member_type: VOTER }
I20240502 14:06:54.250689 16823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 854848cec8144d36b0e06dd804df9293 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:06:54.250882 16823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 854848cec8144d36b0e06dd804df9293 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:06:54.251084 16823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 854848cec8144d36b0e06dd804df9293 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:54.255110 16823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 854848cec8144d36b0e06dd804df9293 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "854848cec8144d36b0e06dd804df9293" member_type: VOTER }
I20240502 14:06:54.255548 16823 leader_election.cc:302] T 00000000000000000000000000000000 P 854848cec8144d36b0e06dd804df9293 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 854848cec8144d36b0e06dd804df9293; no voters:
I20240502 14:06:54.256474 16823 leader_election.cc:288] T 00000000000000000000000000000000 P 854848cec8144d36b0e06dd804df9293 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:06:54.256803 16826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 854848cec8144d36b0e06dd804df9293 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:54.258185 16826 raft_consensus.cc:686] T 00000000000000000000000000000000 P 854848cec8144d36b0e06dd804df9293 [term 1 LEADER]: Becoming Leader. State: Replica: 854848cec8144d36b0e06dd804df9293, State: Running, Role: LEADER
I20240502 14:06:54.258796 16826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 854848cec8144d36b0e06dd804df9293 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "854848cec8144d36b0e06dd804df9293" member_type: VOTER }
I20240502 14:06:54.259301 16823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 854848cec8144d36b0e06dd804df9293 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:06:54.263612 16828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 854848cec8144d36b0e06dd804df9293 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 854848cec8144d36b0e06dd804df9293. Latest consensus state: current_term: 1 leader_uuid: "854848cec8144d36b0e06dd804df9293" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "854848cec8144d36b0e06dd804df9293" member_type: VOTER } }
I20240502 14:06:54.264240 16828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 854848cec8144d36b0e06dd804df9293 [sys.catalog]: This master's current role is: LEADER
I20240502 14:06:54.264504 16827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 854848cec8144d36b0e06dd804df9293 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "854848cec8144d36b0e06dd804df9293" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "854848cec8144d36b0e06dd804df9293" member_type: VOTER } }
I20240502 14:06:54.264979 16827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 854848cec8144d36b0e06dd804df9293 [sys.catalog]: This master's current role is: LEADER
I20240502 14:06:54.266232 16833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:06:54.270491 16833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:06:54.275132 11301 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 14:06:54.278183 16833 catalog_manager.cc:1347] Generated new cluster ID: 1636c4f8a77043cea23090ce8896b364
I20240502 14:06:54.278452 16833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:06:54.296330 16833 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:06:54.297638 16833 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:06:54.312511 16833 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 854848cec8144d36b0e06dd804df9293: Generated new TSK 0
I20240502 14:06:54.313097 16833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:06:54.341482 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:54.346834 16844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:54.348481 16845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:54.349680 16847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:54.350348 11301 server_base.cc:1034] running on GCE node
I20240502 14:06:54.351120 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:06:54.351294 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:54.351455 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658814351435 us; error 0 us; skew 500 ppm
I20240502 14:06:54.351951 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:54.354244 11301 webserver.cc:460] Webserver started at http://127.11.9.65:33675/ using document root <none> and password file <none>
I20240502 14:06:54.354686 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:54.354851 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:54.355084 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:54.356134 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-0-root/instance:
uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d"
format_stamp: "Formatted at 2024-05-02 14:06:54 on dist-test-slave-j72s"
I20240502 14:06:54.360271 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 14:06:54.363308 16852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:54.363970 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:06:54.364240 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-0-root
uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d"
format_stamp: "Formatted at 2024-05-02 14:06:54 on dist-test-slave-j72s"
I20240502 14:06:54.364490 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:54.386994 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:54.387984 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:54.389283 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:54.391240 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:54.391427 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:54.391635 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:54.391798 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:54.428921 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.65:44993
I20240502 14:06:54.429018 16914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.65:44993 every 8 connection(s)
I20240502 14:06:54.433315 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:54.441121 16919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:54.442727 16920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:54.444406 16922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:54.444895 16915 heartbeater.cc:347] Connected to a master server at 127.11.9.126:36895
I20240502 14:06:54.445235 16915 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:54.445832 11301 server_base.cc:1034] running on GCE node
I20240502 14:06:54.446122 16915 heartbeater.cc:511] Master 127.11.9.126:36895 requested a full tablet report, sending...
I20240502 14:06:54.446939 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:06:54.447196 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:54.447389 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658814447365 us; error 0 us; skew 500 ppm
I20240502 14:06:54.448072 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:54.448382 16788 ts_manager.cc:194] Registered new tserver with Master: b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993)
I20240502 14:06:54.450286 16788 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48634
I20240502 14:06:54.450558 11301 webserver.cc:460] Webserver started at http://127.11.9.66:38861/ using document root <none> and password file <none>
I20240502 14:06:54.451084 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:54.451376 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:54.451701 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:54.453047 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-1-root/instance:
uuid: "38ec2862e51846ce84552fa0e7f025f2"
format_stamp: "Formatted at 2024-05-02 14:06:54 on dist-test-slave-j72s"
I20240502 14:06:54.457345 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:06:54.460528 16927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:54.461244 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:06:54.461539 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-1-root
uuid: "38ec2862e51846ce84552fa0e7f025f2"
format_stamp: "Formatted at 2024-05-02 14:06:54 on dist-test-slave-j72s"
I20240502 14:06:54.461814 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:54.479022 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:54.480098 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:54.481408 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:54.483655 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:54.483836 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:54.484050 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:54.484198 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:54.520856 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.66:35709
I20240502 14:06:54.520953 16989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.66:35709 every 8 connection(s)
I20240502 14:06:54.525281 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:54.531373 16993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:54.532418 16994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:54.533722 16996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:54.534988 11301 server_base.cc:1034] running on GCE node
I20240502 14:06:54.535864 16990 heartbeater.cc:347] Connected to a master server at 127.11.9.126:36895
I20240502 14:06:54.535877 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 14:06:54.536226 16990 heartbeater.cc:464] Registering TS with master...
W20240502 14:06:54.536235 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:54.536605 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658814536590 us; error 0 us; skew 500 ppm
I20240502 14:06:54.537010 16990 heartbeater.cc:511] Master 127.11.9.126:36895 requested a full tablet report, sending...
I20240502 14:06:54.537129 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:54.539127 16788 ts_manager.cc:194] Registered new tserver with Master: 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66:35709)
I20240502 14:06:54.540015 11301 webserver.cc:460] Webserver started at http://127.11.9.67:33439/ using document root <none> and password file <none>
I20240502 14:06:54.540616 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:54.540836 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:54.540920 16788 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48650
I20240502 14:06:54.541219 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:54.542449 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-2-root/instance:
uuid: "d4d180df6a2f4d78b0296a096b748a71"
format_stamp: "Formatted at 2024-05-02 14:06:54 on dist-test-slave-j72s"
I20240502 14:06:54.546361 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240502 14:06:54.549240 17001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:54.549994 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:06:54.550243 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-2-root
uuid: "d4d180df6a2f4d78b0296a096b748a71"
format_stamp: "Formatted at 2024-05-02 14:06:54 on dist-test-slave-j72s"
I20240502 14:06:54.550460 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:54.562628 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:54.563500 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:54.564726 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:54.566741 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:54.566905 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:54.567124 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:54.567251 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:54.603603 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.67:35327
I20240502 14:06:54.603699 17063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.67:35327 every 8 connection(s)
I20240502 14:06:54.615345 17064 heartbeater.cc:347] Connected to a master server at 127.11.9.126:36895
I20240502 14:06:54.615638 17064 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:54.616170 17064 heartbeater.cc:511] Master 127.11.9.126:36895 requested a full tablet report, sending...
I20240502 14:06:54.617820 16788 ts_manager.cc:194] Registered new tserver with Master: d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327)
I20240502 14:06:54.618366 11301 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012053203s
I20240502 14:06:54.619098 16788 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49960
I20240502 14:06:55.453068 16915 heartbeater.cc:503] Master 127.11.9.126:36895 was elected leader, sending a full tablet report...
I20240502 14:06:55.543427 16990 heartbeater.cc:503] Master 127.11.9.126:36895 was elected leader, sending a full tablet report...
I20240502 14:06:55.621690 17064 heartbeater.cc:503] Master 127.11.9.126:36895 was elected leader, sending a full tablet report...
I20240502 14:06:55.650043 11301 test_util.cc:268] Using random seed: 496904416
I20240502 14:06:55.678031 16788 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49968:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:06:55.680357 16788 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:06:55.740401 16955 tablet_service.cc:1467] Processing CreateTablet for tablet c6742043c1e74f269ea157de517c5e98 (DEFAULT_TABLE table=test-workload [id=7c460605d9c1462d9bb93b606e63b679]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 14:06:55.741833 16955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6742043c1e74f269ea157de517c5e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:55.743062 16953 tablet_service.cc:1467] Processing CreateTablet for tablet 94012a9173e74331942e131aeb26320c (DEFAULT_TABLE table=test-workload [id=7c460605d9c1462d9bb93b606e63b679]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 14:06:55.744323 16953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94012a9173e74331942e131aeb26320c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:55.750639 16952 tablet_service.cc:1467] Processing CreateTablet for tablet dd932332efd64937b1142facd80b7be0 (DEFAULT_TABLE table=test-workload [id=7c460605d9c1462d9bb93b606e63b679]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 14:06:55.751029 17029 tablet_service.cc:1467] Processing CreateTablet for tablet c6742043c1e74f269ea157de517c5e98 (DEFAULT_TABLE table=test-workload [id=7c460605d9c1462d9bb93b606e63b679]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 14:06:55.751430 17028 tablet_service.cc:1467] Processing CreateTablet for tablet f476570d1ef4476ba22a79694a340e88 (DEFAULT_TABLE table=test-workload [id=7c460605d9c1462d9bb93b606e63b679]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 14:06:55.752637 17029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6742043c1e74f269ea157de517c5e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:55.753221 17028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f476570d1ef4476ba22a79694a340e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:55.754946 16877 tablet_service.cc:1467] Processing CreateTablet for tablet dd932332efd64937b1142facd80b7be0 (DEFAULT_TABLE table=test-workload [id=7c460605d9c1462d9bb93b606e63b679]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 14:06:55.756170 16877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd932332efd64937b1142facd80b7be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:55.758562 16878 tablet_service.cc:1467] Processing CreateTablet for tablet 94012a9173e74331942e131aeb26320c (DEFAULT_TABLE table=test-workload [id=7c460605d9c1462d9bb93b606e63b679]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 14:06:55.760807 16880 tablet_service.cc:1467] Processing CreateTablet for tablet c6742043c1e74f269ea157de517c5e98 (DEFAULT_TABLE table=test-workload [id=7c460605d9c1462d9bb93b606e63b679]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 14:06:55.762125 16880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6742043c1e74f269ea157de517c5e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:55.762487 16879 tablet_service.cc:1467] Processing CreateTablet for tablet f476570d1ef4476ba22a79694a340e88 (DEFAULT_TABLE table=test-workload [id=7c460605d9c1462d9bb93b606e63b679]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 14:06:55.763727 16879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f476570d1ef4476ba22a79694a340e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:55.741446 16954 tablet_service.cc:1467] Processing CreateTablet for tablet f476570d1ef4476ba22a79694a340e88 (DEFAULT_TABLE table=test-workload [id=7c460605d9c1462d9bb93b606e63b679]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 14:06:55.766270 17027 tablet_service.cc:1467] Processing CreateTablet for tablet 94012a9173e74331942e131aeb26320c (DEFAULT_TABLE table=test-workload [id=7c460605d9c1462d9bb93b606e63b679]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 14:06:55.767300 16878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94012a9173e74331942e131aeb26320c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:55.767565 17027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94012a9173e74331942e131aeb26320c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:55.766279 16952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd932332efd64937b1142facd80b7be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:55.773869 16954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f476570d1ef4476ba22a79694a340e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:55.780401 17026 tablet_service.cc:1467] Processing CreateTablet for tablet dd932332efd64937b1142facd80b7be0 (DEFAULT_TABLE table=test-workload [id=7c460605d9c1462d9bb93b606e63b679]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 14:06:55.782023 17026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd932332efd64937b1142facd80b7be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:55.813797 17084 tablet_bootstrap.cc:492] T c6742043c1e74f269ea157de517c5e98 P d4d180df6a2f4d78b0296a096b748a71: Bootstrap starting.
I20240502 14:06:55.819535 17085 tablet_bootstrap.cc:492] T f476570d1ef4476ba22a79694a340e88 P 38ec2862e51846ce84552fa0e7f025f2: Bootstrap starting.
I20240502 14:06:55.827008 17086 tablet_bootstrap.cc:492] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap starting.
I20240502 14:06:55.819681 17084 tablet_bootstrap.cc:654] T c6742043c1e74f269ea157de517c5e98 P d4d180df6a2f4d78b0296a096b748a71: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:55.832419 17085 tablet_bootstrap.cc:654] T f476570d1ef4476ba22a79694a340e88 P 38ec2862e51846ce84552fa0e7f025f2: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:55.832993 17086 tablet_bootstrap.cc:654] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:55.833792 17084 tablet_bootstrap.cc:492] T c6742043c1e74f269ea157de517c5e98 P d4d180df6a2f4d78b0296a096b748a71: No bootstrap required, opened a new log
I20240502 14:06:55.834290 17084 ts_tablet_manager.cc:1367] T c6742043c1e74f269ea157de517c5e98 P d4d180df6a2f4d78b0296a096b748a71: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.004s
I20240502 14:06:55.837090 17084 raft_consensus.cc:348] T c6742043c1e74f269ea157de517c5e98 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } } peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } }
I20240502 14:06:55.837622 17086 tablet_bootstrap.cc:492] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d: No bootstrap required, opened a new log
I20240502 14:06:55.837920 17084 raft_consensus.cc:374] T c6742043c1e74f269ea157de517c5e98 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:55.837994 17085 tablet_bootstrap.cc:492] T f476570d1ef4476ba22a79694a340e88 P 38ec2862e51846ce84552fa0e7f025f2: No bootstrap required, opened a new log
I20240502 14:06:55.838279 17086 ts_tablet_manager.cc:1367] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.011s
I20240502 14:06:55.838251 17084 raft_consensus.cc:729] T c6742043c1e74f269ea157de517c5e98 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4d180df6a2f4d78b0296a096b748a71, State: Initialized, Role: FOLLOWER
I20240502 14:06:55.838583 17085 ts_tablet_manager.cc:1367] T f476570d1ef4476ba22a79694a340e88 P 38ec2862e51846ce84552fa0e7f025f2: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.006s
I20240502 14:06:55.839285 17084 consensus_queue.cc:260] T c6742043c1e74f269ea157de517c5e98 P d4d180df6a2f4d78b0296a096b748a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } } peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } }
I20240502 14:06:55.841347 17085 raft_consensus.cc:348] T f476570d1ef4476ba22a79694a340e88 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:55.842113 17085 raft_consensus.cc:374] T f476570d1ef4476ba22a79694a340e88 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:55.842240 17084 ts_tablet_manager.cc:1398] T c6742043c1e74f269ea157de517c5e98 P d4d180df6a2f4d78b0296a096b748a71: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240502 14:06:55.842433 17085 raft_consensus.cc:729] T f476570d1ef4476ba22a79694a340e88 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ec2862e51846ce84552fa0e7f025f2, State: Initialized, Role: FOLLOWER
I20240502 14:06:55.843142 17084 tablet_bootstrap.cc:492] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71: Bootstrap starting.
I20240502 14:06:55.843147 17085 consensus_queue.cc:260] T f476570d1ef4476ba22a79694a340e88 P 38ec2862e51846ce84552fa0e7f025f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:55.849200 17084 tablet_bootstrap.cc:654] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:55.850591 17086 raft_consensus.cc:348] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } } peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } }
I20240502 14:06:55.851387 17086 raft_consensus.cc:374] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:55.851644 17086 raft_consensus.cc:729] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e9c2da7a3b4267b3365a1f23bfb17d, State: Initialized, Role: FOLLOWER
I20240502 14:06:55.852774 17086 consensus_queue.cc:260] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } } peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } }
I20240502 14:06:55.858831 17086 ts_tablet_manager.cc:1398] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent starting tablet: real 0.020s user 0.008s sys 0.001s
I20240502 14:06:55.862782 17086 tablet_bootstrap.cc:492] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap starting.
I20240502 14:06:55.870199 17085 ts_tablet_manager.cc:1398] T f476570d1ef4476ba22a79694a340e88 P 38ec2862e51846ce84552fa0e7f025f2: Time spent starting tablet: real 0.031s user 0.021s sys 0.008s
I20240502 14:06:55.870594 17084 tablet_bootstrap.cc:492] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71: No bootstrap required, opened a new log
I20240502 14:06:55.871074 17084 ts_tablet_manager.cc:1367] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.011s
I20240502 14:06:55.871186 17086 tablet_bootstrap.cc:654] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:55.875272 17084 raft_consensus.cc:348] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:55.876009 17084 raft_consensus.cc:374] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:55.876299 17084 raft_consensus.cc:729] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4d180df6a2f4d78b0296a096b748a71, State: Initialized, Role: FOLLOWER
I20240502 14:06:55.876801 17085 tablet_bootstrap.cc:492] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2: Bootstrap starting.
I20240502 14:06:55.877095 17084 consensus_queue.cc:260] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:55.879244 17084 ts_tablet_manager.cc:1398] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240502 14:06:55.880225 17084 tablet_bootstrap.cc:492] T 94012a9173e74331942e131aeb26320c P d4d180df6a2f4d78b0296a096b748a71: Bootstrap starting.
I20240502 14:06:55.882732 17085 tablet_bootstrap.cc:654] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:55.885774 17084 tablet_bootstrap.cc:654] T 94012a9173e74331942e131aeb26320c P d4d180df6a2f4d78b0296a096b748a71: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:55.891148 17084 tablet_bootstrap.cc:492] T 94012a9173e74331942e131aeb26320c P d4d180df6a2f4d78b0296a096b748a71: No bootstrap required, opened a new log
I20240502 14:06:55.891635 17084 ts_tablet_manager.cc:1367] T 94012a9173e74331942e131aeb26320c P d4d180df6a2f4d78b0296a096b748a71: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20240502 14:06:55.892503 17086 tablet_bootstrap.cc:492] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d: No bootstrap required, opened a new log
I20240502 14:06:55.892984 17086 ts_tablet_manager.cc:1367] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.002s
I20240502 14:06:55.894446 17084 raft_consensus.cc:348] T 94012a9173e74331942e131aeb26320c P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:55.895252 17084 raft_consensus.cc:374] T 94012a9173e74331942e131aeb26320c P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:55.895568 17084 raft_consensus.cc:729] T 94012a9173e74331942e131aeb26320c P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4d180df6a2f4d78b0296a096b748a71, State: Initialized, Role: FOLLOWER
I20240502 14:06:55.895925 17086 raft_consensus.cc:348] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:55.896682 17086 raft_consensus.cc:374] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:55.896384 17084 consensus_queue.cc:260] T 94012a9173e74331942e131aeb26320c P d4d180df6a2f4d78b0296a096b748a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:55.896984 17086 raft_consensus.cc:729] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e9c2da7a3b4267b3365a1f23bfb17d, State: Initialized, Role: FOLLOWER
I20240502 14:06:55.897747 17085 tablet_bootstrap.cc:492] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2: No bootstrap required, opened a new log
I20240502 14:06:55.898216 17085 ts_tablet_manager.cc:1367] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.004s
I20240502 14:06:55.899037 17084 ts_tablet_manager.cc:1398] T 94012a9173e74331942e131aeb26320c P d4d180df6a2f4d78b0296a096b748a71: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20240502 14:06:55.900071 17084 tablet_bootstrap.cc:492] T dd932332efd64937b1142facd80b7be0 P d4d180df6a2f4d78b0296a096b748a71: Bootstrap starting.
I20240502 14:06:55.900573 17085 raft_consensus.cc:348] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } } peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } }
I20240502 14:06:55.901226 17085 raft_consensus.cc:374] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:55.901475 17085 raft_consensus.cc:729] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ec2862e51846ce84552fa0e7f025f2, State: Initialized, Role: FOLLOWER
I20240502 14:06:55.902097 17085 consensus_queue.cc:260] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } } peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } }
I20240502 14:06:55.906278 17086 consensus_queue.cc:260] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:55.909713 17085 ts_tablet_manager.cc:1398] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2: Time spent starting tablet: real 0.011s user 0.004s sys 0.002s
I20240502 14:06:55.910229 17086 ts_tablet_manager.cc:1398] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent starting tablet: real 0.017s user 0.006s sys 0.004s
I20240502 14:06:55.911656 17086 tablet_bootstrap.cc:492] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap starting.
I20240502 14:06:55.906282 17084 tablet_bootstrap.cc:654] T dd932332efd64937b1142facd80b7be0 P d4d180df6a2f4d78b0296a096b748a71: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:55.915351 17085 tablet_bootstrap.cc:492] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2: Bootstrap starting.
I20240502 14:06:55.918661 17086 tablet_bootstrap.cc:654] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:55.921774 17085 tablet_bootstrap.cc:654] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:55.928861 17086 tablet_bootstrap.cc:492] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d: No bootstrap required, opened a new log
I20240502 14:06:55.929359 17086 ts_tablet_manager.cc:1367] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.006s
I20240502 14:06:55.929952 17084 tablet_bootstrap.cc:492] T dd932332efd64937b1142facd80b7be0 P d4d180df6a2f4d78b0296a096b748a71: No bootstrap required, opened a new log
I20240502 14:06:55.930449 17084 ts_tablet_manager.cc:1367] T dd932332efd64937b1142facd80b7be0 P d4d180df6a2f4d78b0296a096b748a71: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.012s
I20240502 14:06:55.931128 17085 tablet_bootstrap.cc:492] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2: No bootstrap required, opened a new log
I20240502 14:06:55.931747 17085 ts_tablet_manager.cc:1367] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.003s
I20240502 14:06:55.933069 17084 raft_consensus.cc:348] T dd932332efd64937b1142facd80b7be0 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:55.931973 17086 raft_consensus.cc:348] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:55.933840 17084 raft_consensus.cc:374] T dd932332efd64937b1142facd80b7be0 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:55.934002 17086 raft_consensus.cc:374] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:55.934204 17084 raft_consensus.cc:729] T dd932332efd64937b1142facd80b7be0 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4d180df6a2f4d78b0296a096b748a71, State: Initialized, Role: FOLLOWER
I20240502 14:06:55.934455 17086 raft_consensus.cc:729] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e9c2da7a3b4267b3365a1f23bfb17d, State: Initialized, Role: FOLLOWER
I20240502 14:06:55.935034 17085 raft_consensus.cc:348] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:55.934993 17084 consensus_queue.cc:260] T dd932332efd64937b1142facd80b7be0 P d4d180df6a2f4d78b0296a096b748a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:55.935206 17086 consensus_queue.cc:260] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:55.935868 17085 raft_consensus.cc:374] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:55.936175 17085 raft_consensus.cc:729] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ec2862e51846ce84552fa0e7f025f2, State: Initialized, Role: FOLLOWER
I20240502 14:06:55.936872 17085 consensus_queue.cc:260] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:55.937357 17084 ts_tablet_manager.cc:1398] T dd932332efd64937b1142facd80b7be0 P d4d180df6a2f4d78b0296a096b748a71: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240502 14:06:55.938767 17085 ts_tablet_manager.cc:1398] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 14:06:55.939702 17085 tablet_bootstrap.cc:492] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2: Bootstrap starting.
I20240502 14:06:55.940749 17086 ts_tablet_manager.cc:1398] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent starting tablet: real 0.011s user 0.001s sys 0.004s
I20240502 14:06:55.941700 17086 tablet_bootstrap.cc:492] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap starting.
I20240502 14:06:55.944960 17085 tablet_bootstrap.cc:654] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:55.947088 17086 tablet_bootstrap.cc:654] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:55.950747 17085 tablet_bootstrap.cc:492] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2: No bootstrap required, opened a new log
I20240502 14:06:55.951382 17085 ts_tablet_manager.cc:1367] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20240502 14:06:55.954187 17085 raft_consensus.cc:348] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:55.954957 17085 raft_consensus.cc:374] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:55.955242 17085 raft_consensus.cc:729] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ec2862e51846ce84552fa0e7f025f2, State: Initialized, Role: FOLLOWER
I20240502 14:06:55.955982 17085 consensus_queue.cc:260] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:55.958922 17085 ts_tablet_manager.cc:1398] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 14:06:55.961707 17086 tablet_bootstrap.cc:492] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d: No bootstrap required, opened a new log
I20240502 14:06:55.962179 17086 ts_tablet_manager.cc:1367] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.007s
I20240502 14:06:55.964206 17086 raft_consensus.cc:348] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:55.964725 17086 raft_consensus.cc:374] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:55.964973 17086 raft_consensus.cc:729] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e9c2da7a3b4267b3365a1f23bfb17d, State: Initialized, Role: FOLLOWER
I20240502 14:06:55.965569 17086 consensus_queue.cc:260] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:55.967197 17086 ts_tablet_manager.cc:1398] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20240502 14:06:55.972206 17093 raft_consensus.cc:482] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:55.972676 17093 raft_consensus.cc:504] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:55.974676 17093 leader_election.cc:288] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66:35709), d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327)
I20240502 14:06:55.986289 17039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd932332efd64937b1142facd80b7be0" candidate_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4d180df6a2f4d78b0296a096b748a71" is_pre_election: true
I20240502 14:06:55.986910 17039 raft_consensus.cc:2454] T dd932332efd64937b1142facd80b7be0 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0e9c2da7a3b4267b3365a1f23bfb17d in term 0.
I20240502 14:06:55.987864 16854 leader_election.cc:302] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0e9c2da7a3b4267b3365a1f23bfb17d, d4d180df6a2f4d78b0296a096b748a71; no voters:
I20240502 14:06:55.988528 17093 raft_consensus.cc:2781] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:55.988853 17093 raft_consensus.cc:482] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:55.989254 17093 raft_consensus.cc:3037] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:55.990653 16965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd932332efd64937b1142facd80b7be0" candidate_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ec2862e51846ce84552fa0e7f025f2" is_pre_election: true
I20240502 14:06:55.991405 16965 raft_consensus.cc:2454] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0e9c2da7a3b4267b3365a1f23bfb17d in term 0.
I20240502 14:06:55.992471 17104 raft_consensus.cc:482] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:55.992869 17104 raft_consensus.cc:504] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:55.994583 17104 leader_election.cc:288] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66:35709), d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327)
I20240502 14:06:55.995364 16965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f476570d1ef4476ba22a79694a340e88" candidate_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ec2862e51846ce84552fa0e7f025f2" is_pre_election: true
I20240502 14:06:55.995417 17093 raft_consensus.cc:504] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:55.995932 17039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f476570d1ef4476ba22a79694a340e88" candidate_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4d180df6a2f4d78b0296a096b748a71" is_pre_election: true
I20240502 14:06:55.996150 16965 raft_consensus.cc:2454] T f476570d1ef4476ba22a79694a340e88 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0e9c2da7a3b4267b3365a1f23bfb17d in term 0.
I20240502 14:06:55.996429 17039 raft_consensus.cc:2454] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0e9c2da7a3b4267b3365a1f23bfb17d in term 0.
I20240502 14:06:55.997388 17093 leader_election.cc:288] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [CANDIDATE]: Term 1 election: Requested vote from peers 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66:35709), d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327)
I20240502 14:06:55.997339 16854 leader_election.cc:302] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0e9c2da7a3b4267b3365a1f23bfb17d, d4d180df6a2f4d78b0296a096b748a71; no voters:
I20240502 14:06:55.998075 16965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd932332efd64937b1142facd80b7be0" candidate_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ec2862e51846ce84552fa0e7f025f2"
I20240502 14:06:55.998425 17093 raft_consensus.cc:2781] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:55.998685 16965 raft_consensus.cc:3037] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:55.998862 17093 raft_consensus.cc:482] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:55.999316 17093 raft_consensus.cc:3037] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:55.999346 17039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd932332efd64937b1142facd80b7be0" candidate_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4d180df6a2f4d78b0296a096b748a71"
I20240502 14:06:56.000022 17039 raft_consensus.cc:3037] T dd932332efd64937b1142facd80b7be0 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:56.003721 16965 raft_consensus.cc:2454] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0e9c2da7a3b4267b3365a1f23bfb17d in term 1.
I20240502 14:06:56.004639 16855 leader_election.cc:302] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ec2862e51846ce84552fa0e7f025f2, b0e9c2da7a3b4267b3365a1f23bfb17d; no voters:
I20240502 14:06:56.005328 17104 raft_consensus.cc:2781] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:56.006042 17093 raft_consensus.cc:504] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.006608 17104 raft_consensus.cc:686] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 LEADER]: Becoming Leader. State: Replica: b0e9c2da7a3b4267b3365a1f23bfb17d, State: Running, Role: LEADER
I20240502 14:06:56.006762 17039 raft_consensus.cc:2454] T dd932332efd64937b1142facd80b7be0 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0e9c2da7a3b4267b3365a1f23bfb17d in term 1.
I20240502 14:06:56.007609 17104 consensus_queue.cc:237] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.008463 17093 leader_election.cc:288] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [CANDIDATE]: Term 1 election: Requested vote from peers 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66:35709), d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327)
I20240502 14:06:56.008746 16965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f476570d1ef4476ba22a79694a340e88" candidate_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ec2862e51846ce84552fa0e7f025f2"
I20240502 14:06:56.009238 16965 raft_consensus.cc:3037] T f476570d1ef4476ba22a79694a340e88 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:56.010624 17039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f476570d1ef4476ba22a79694a340e88" candidate_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4d180df6a2f4d78b0296a096b748a71"
I20240502 14:06:56.013757 17039 raft_consensus.cc:3037] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:56.015080 16965 raft_consensus.cc:2454] T f476570d1ef4476ba22a79694a340e88 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0e9c2da7a3b4267b3365a1f23bfb17d in term 1.
I20240502 14:06:56.016114 16855 leader_election.cc:302] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ec2862e51846ce84552fa0e7f025f2, b0e9c2da7a3b4267b3365a1f23bfb17d; no voters:
I20240502 14:06:56.016901 17104 raft_consensus.cc:2781] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:56.016451 16788 catalog_manager.cc:5484] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d reported cstate change: term changed from 0 to 1, leader changed from <none> to b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65). New cstate: current_term: 1 leader_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:56.017593 17104 raft_consensus.cc:686] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 LEADER]: Becoming Leader. State: Replica: b0e9c2da7a3b4267b3365a1f23bfb17d, State: Running, Role: LEADER
I20240502 14:06:56.018396 17104 consensus_queue.cc:237] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.019578 17039 raft_consensus.cc:2454] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0e9c2da7a3b4267b3365a1f23bfb17d in term 1.
I20240502 14:06:56.028517 16788 catalog_manager.cc:5484] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d reported cstate change: term changed from 0 to 1, leader changed from <none> to b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65). New cstate: current_term: 1 leader_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:56.063794 17104 raft_consensus.cc:482] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:56.064296 17104 raft_consensus.cc:504] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.066011 17104 leader_election.cc:288] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66:35709), d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327)
I20240502 14:06:56.066825 16965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94012a9173e74331942e131aeb26320c" candidate_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ec2862e51846ce84552fa0e7f025f2" is_pre_election: true
I20240502 14:06:56.067004 17039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94012a9173e74331942e131aeb26320c" candidate_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4d180df6a2f4d78b0296a096b748a71" is_pre_election: true
I20240502 14:06:56.067538 16965 raft_consensus.cc:2454] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0e9c2da7a3b4267b3365a1f23bfb17d in term 0.
I20240502 14:06:56.067550 17039 raft_consensus.cc:2454] T 94012a9173e74331942e131aeb26320c P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0e9c2da7a3b4267b3365a1f23bfb17d in term 0.
I20240502 14:06:56.068794 16854 leader_election.cc:302] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0e9c2da7a3b4267b3365a1f23bfb17d, d4d180df6a2f4d78b0296a096b748a71; no voters:
I20240502 14:06:56.069439 17104 raft_consensus.cc:2781] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:56.069840 17104 raft_consensus.cc:482] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:56.070093 17104 raft_consensus.cc:3037] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:56.074816 17104 raft_consensus.cc:504] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.076396 17104 leader_election.cc:288] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d [CANDIDATE]: Term 1 election: Requested vote from peers 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66:35709), d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327)
I20240502 14:06:56.077165 16965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94012a9173e74331942e131aeb26320c" candidate_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ec2862e51846ce84552fa0e7f025f2"
I20240502 14:06:56.077270 17039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94012a9173e74331942e131aeb26320c" candidate_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4d180df6a2f4d78b0296a096b748a71"
I20240502 14:06:56.077723 16965 raft_consensus.cc:3037] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:56.077807 17039 raft_consensus.cc:3037] T 94012a9173e74331942e131aeb26320c P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:56.082201 17039 raft_consensus.cc:2454] T 94012a9173e74331942e131aeb26320c P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0e9c2da7a3b4267b3365a1f23bfb17d in term 1.
I20240502 14:06:56.082207 16965 raft_consensus.cc:2454] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0e9c2da7a3b4267b3365a1f23bfb17d in term 1.
I20240502 14:06:56.083482 16854 leader_election.cc:302] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0e9c2da7a3b4267b3365a1f23bfb17d, d4d180df6a2f4d78b0296a096b748a71; no voters:
I20240502 14:06:56.084203 17104 raft_consensus.cc:2781] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:56.084676 17104 raft_consensus.cc:686] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 LEADER]: Becoming Leader. State: Replica: b0e9c2da7a3b4267b3365a1f23bfb17d, State: Running, Role: LEADER
I20240502 14:06:56.085395 17104 consensus_queue.cc:237] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.092201 16788 catalog_manager.cc:5484] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d reported cstate change: term changed from 0 to 1, leader changed from <none> to b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65). New cstate: current_term: 1 leader_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:56.246641 17104 raft_consensus.cc:482] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:56.247107 17104 raft_consensus.cc:504] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } } peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } }
I20240502 14:06:56.248704 17104 leader_election.cc:288] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66:35709), d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327)
I20240502 14:06:56.249622 16965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6742043c1e74f269ea157de517c5e98" candidate_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ec2862e51846ce84552fa0e7f025f2" is_pre_election: true
I20240502 14:06:56.249722 17039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6742043c1e74f269ea157de517c5e98" candidate_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4d180df6a2f4d78b0296a096b748a71" is_pre_election: true
I20240502 14:06:56.250188 16965 raft_consensus.cc:2454] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0e9c2da7a3b4267b3365a1f23bfb17d in term 0.
I20240502 14:06:56.250257 17039 raft_consensus.cc:2454] T c6742043c1e74f269ea157de517c5e98 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0e9c2da7a3b4267b3365a1f23bfb17d in term 0.
I20240502 14:06:56.251120 16855 leader_election.cc:302] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ec2862e51846ce84552fa0e7f025f2, b0e9c2da7a3b4267b3365a1f23bfb17d; no voters:
I20240502 14:06:56.251860 17104 raft_consensus.cc:2781] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:56.252197 17104 raft_consensus.cc:482] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:56.252506 17104 raft_consensus.cc:3037] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:56.256803 17104 raft_consensus.cc:504] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } } peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } }
I20240502 14:06:56.258189 17104 leader_election.cc:288] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [CANDIDATE]: Term 1 election: Requested vote from peers 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66:35709), d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327)
I20240502 14:06:56.258919 16965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6742043c1e74f269ea157de517c5e98" candidate_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ec2862e51846ce84552fa0e7f025f2"
I20240502 14:06:56.259122 17039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6742043c1e74f269ea157de517c5e98" candidate_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4d180df6a2f4d78b0296a096b748a71"
I20240502 14:06:56.259465 16965 raft_consensus.cc:3037] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:56.259605 17039 raft_consensus.cc:3037] T c6742043c1e74f269ea157de517c5e98 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:56.264065 17039 raft_consensus.cc:2454] T c6742043c1e74f269ea157de517c5e98 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0e9c2da7a3b4267b3365a1f23bfb17d in term 1.
I20240502 14:06:56.264058 16965 raft_consensus.cc:2454] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0e9c2da7a3b4267b3365a1f23bfb17d in term 1.
I20240502 14:06:56.265050 16854 leader_election.cc:302] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0e9c2da7a3b4267b3365a1f23bfb17d, d4d180df6a2f4d78b0296a096b748a71; no voters:
I20240502 14:06:56.265770 17104 raft_consensus.cc:2781] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:56.266165 17104 raft_consensus.cc:686] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 LEADER]: Becoming Leader. State: Replica: b0e9c2da7a3b4267b3365a1f23bfb17d, State: Running, Role: LEADER
I20240502 14:06:56.266866 17104 consensus_queue.cc:237] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } } peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } }
W20240502 14:06:56.271147 16836 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 14:06:56.273414 16788 catalog_manager.cc:5484] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d reported cstate change: term changed from 0 to 1, leader changed from <none> to b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65). New cstate: current_term: 1 leader_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } health_report { overall_health: HEALTHY } } }
I20240502 14:06:56.301559 11301 test_util.cc:268] Using random seed: 497555887
I20240502 14:06:56.321796 16788 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49974:
name: "dugtrio"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 14:06:56.323798 16788 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table dugtrio in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 14:06:56.350524 16880 tablet_service.cc:1467] Processing CreateTablet for tablet 47ce8743e49b4ef68a2ddaa7a9a55a76 (DEFAULT_TABLE table=dugtrio [id=319ac21d405a474db2378120a46398fb]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 14:06:56.351095 16953 tablet_service.cc:1467] Processing CreateTablet for tablet 47ce8743e49b4ef68a2ddaa7a9a55a76 (DEFAULT_TABLE table=dugtrio [id=319ac21d405a474db2378120a46398fb]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 14:06:56.351764 16880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ce8743e49b4ef68a2ddaa7a9a55a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:56.352316 16953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ce8743e49b4ef68a2ddaa7a9a55a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:56.353133 17026 tablet_service.cc:1467] Processing CreateTablet for tablet 47ce8743e49b4ef68a2ddaa7a9a55a76 (DEFAULT_TABLE table=dugtrio [id=319ac21d405a474db2378120a46398fb]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240502 14:06:56.354322 17026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ce8743e49b4ef68a2ddaa7a9a55a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:56.355713 16954 tablet_service.cc:1467] Processing CreateTablet for tablet 68fb7e1190b949d98edd9f015142ff58 (DEFAULT_TABLE table=dugtrio [id=319ac21d405a474db2378120a46398fb]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 14:06:56.356877 16954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68fb7e1190b949d98edd9f015142ff58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:56.359592 16879 tablet_service.cc:1467] Processing CreateTablet for tablet e5ed65313b9b45eb86098f6208983522 (DEFAULT_TABLE table=dugtrio [id=319ac21d405a474db2378120a46398fb]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 14:06:56.361030 16879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ed65313b9b45eb86098f6208983522. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:56.359596 16877 tablet_service.cc:1467] Processing CreateTablet for tablet 4b68452f8a34439f8cd577bb4be62596 (DEFAULT_TABLE table=dugtrio [id=319ac21d405a474db2378120a46398fb]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 14:06:56.362551 16877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b68452f8a34439f8cd577bb4be62596. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:56.361395 16878 tablet_service.cc:1467] Processing CreateTablet for tablet 68fb7e1190b949d98edd9f015142ff58 (DEFAULT_TABLE table=dugtrio [id=319ac21d405a474db2378120a46398fb]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 14:06:56.363159 16955 tablet_service.cc:1467] Processing CreateTablet for tablet 4b68452f8a34439f8cd577bb4be62596 (DEFAULT_TABLE table=dugtrio [id=319ac21d405a474db2378120a46398fb]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 14:06:56.363701 16878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68fb7e1190b949d98edd9f015142ff58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:56.364334 16955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b68452f8a34439f8cd577bb4be62596. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:56.364123 17027 tablet_service.cc:1467] Processing CreateTablet for tablet 68fb7e1190b949d98edd9f015142ff58 (DEFAULT_TABLE table=dugtrio [id=319ac21d405a474db2378120a46398fb]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240502 14:06:56.365291 17027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68fb7e1190b949d98edd9f015142ff58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:56.367070 17029 tablet_service.cc:1467] Processing CreateTablet for tablet e5ed65313b9b45eb86098f6208983522 (DEFAULT_TABLE table=dugtrio [id=319ac21d405a474db2378120a46398fb]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 14:06:56.368232 17029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ed65313b9b45eb86098f6208983522. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:56.368790 17028 tablet_service.cc:1467] Processing CreateTablet for tablet 4b68452f8a34439f8cd577bb4be62596 (DEFAULT_TABLE table=dugtrio [id=319ac21d405a474db2378120a46398fb]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240502 14:06:56.370009 17028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b68452f8a34439f8cd577bb4be62596. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:56.370455 16952 tablet_service.cc:1467] Processing CreateTablet for tablet e5ed65313b9b45eb86098f6208983522 (DEFAULT_TABLE table=dugtrio [id=319ac21d405a474db2378120a46398fb]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240502 14:06:56.371570 16952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ed65313b9b45eb86098f6208983522. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:56.388793 17086 tablet_bootstrap.cc:492] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap starting.
I20240502 14:06:56.395300 17104 consensus_queue.cc:1035] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:06:56.397476 17084 tablet_bootstrap.cc:492] T 68fb7e1190b949d98edd9f015142ff58 P d4d180df6a2f4d78b0296a096b748a71: Bootstrap starting.
I20240502 14:06:56.397768 17086 tablet_bootstrap.cc:654] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:56.403825 17084 tablet_bootstrap.cc:654] T 68fb7e1190b949d98edd9f015142ff58 P d4d180df6a2f4d78b0296a096b748a71: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:56.412181 17085 tablet_bootstrap.cc:492] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2: Bootstrap starting.
I20240502 14:06:56.417579 17085 tablet_bootstrap.cc:654] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:56.426141 17085 tablet_bootstrap.cc:492] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2: No bootstrap required, opened a new log
I20240502 14:06:56.426599 17085 ts_tablet_manager.cc:1367] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.001s
I20240502 14:06:56.434983 17085 raft_consensus.cc:348] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.435734 17085 raft_consensus.cc:374] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:56.436057 17085 raft_consensus.cc:729] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ec2862e51846ce84552fa0e7f025f2, State: Initialized, Role: FOLLOWER
I20240502 14:06:56.437280 17085 consensus_queue.cc:260] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.439301 17084 tablet_bootstrap.cc:492] T 68fb7e1190b949d98edd9f015142ff58 P d4d180df6a2f4d78b0296a096b748a71: No bootstrap required, opened a new log
I20240502 14:06:56.440058 17084 ts_tablet_manager.cc:1367] T 68fb7e1190b949d98edd9f015142ff58 P d4d180df6a2f4d78b0296a096b748a71: Time spent bootstrapping tablet: real 0.043s user 0.006s sys 0.007s
I20240502 14:06:56.445168 17084 raft_consensus.cc:348] T 68fb7e1190b949d98edd9f015142ff58 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.446252 17084 raft_consensus.cc:374] T 68fb7e1190b949d98edd9f015142ff58 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:56.446694 17084 raft_consensus.cc:729] T 68fb7e1190b949d98edd9f015142ff58 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4d180df6a2f4d78b0296a096b748a71, State: Initialized, Role: FOLLOWER
I20240502 14:06:56.446616 17105 consensus_queue.cc:1035] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:06:56.448808 17085 ts_tablet_manager.cc:1398] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2: Time spent starting tablet: real 0.022s user 0.005s sys 0.003s
I20240502 14:06:56.449496 17084 consensus_queue.cc:260] T 68fb7e1190b949d98edd9f015142ff58 P d4d180df6a2f4d78b0296a096b748a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.457026 17084 ts_tablet_manager.cc:1398] T 68fb7e1190b949d98edd9f015142ff58 P d4d180df6a2f4d78b0296a096b748a71: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20240502 14:06:56.457502 17085 tablet_bootstrap.cc:492] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2: Bootstrap starting.
I20240502 14:06:56.458427 17084 tablet_bootstrap.cc:492] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P d4d180df6a2f4d78b0296a096b748a71: Bootstrap starting.
I20240502 14:06:56.463905 17086 tablet_bootstrap.cc:492] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d: No bootstrap required, opened a new log
I20240502 14:06:56.464473 17086 ts_tablet_manager.cc:1367] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.009s
I20240502 14:06:56.466748 17084 tablet_bootstrap.cc:654] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P d4d180df6a2f4d78b0296a096b748a71: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:56.467504 17085 tablet_bootstrap.cc:654] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:56.472333 17086 raft_consensus.cc:348] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.474418 17105 consensus_queue.cc:1035] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:06:56.482550 17085 tablet_bootstrap.cc:492] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2: No bootstrap required, opened a new log
I20240502 14:06:56.483300 17085 ts_tablet_manager.cc:1367] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.003s
I20240502 14:06:56.473347 17086 raft_consensus.cc:374] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:56.486270 17086 raft_consensus.cc:729] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e9c2da7a3b4267b3365a1f23bfb17d, State: Initialized, Role: FOLLOWER
I20240502 14:06:56.491925 17085 raft_consensus.cc:348] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.492909 17085 raft_consensus.cc:374] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:56.495532 17084 tablet_bootstrap.cc:492] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P d4d180df6a2f4d78b0296a096b748a71: No bootstrap required, opened a new log
I20240502 14:06:56.498556 17085 raft_consensus.cc:729] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ec2862e51846ce84552fa0e7f025f2, State: Initialized, Role: FOLLOWER
I20240502 14:06:56.498908 17084 ts_tablet_manager.cc:1367] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P d4d180df6a2f4d78b0296a096b748a71: Time spent bootstrapping tablet: real 0.041s user 0.014s sys 0.000s
I20240502 14:06:56.499435 17085 consensus_queue.cc:260] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.501706 17085 ts_tablet_manager.cc:1398] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20240502 14:06:56.501458 17084 raft_consensus.cc:348] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.502331 17084 raft_consensus.cc:374] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:56.502619 17085 tablet_bootstrap.cc:492] T 4b68452f8a34439f8cd577bb4be62596 P 38ec2862e51846ce84552fa0e7f025f2: Bootstrap starting.
I20240502 14:06:56.502607 17084 raft_consensus.cc:729] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4d180df6a2f4d78b0296a096b748a71, State: Initialized, Role: FOLLOWER
I20240502 14:06:56.489437 17105 consensus_queue.cc:1035] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20240502 14:06:56.503682 17084 consensus_queue.cc:260] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P d4d180df6a2f4d78b0296a096b748a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.506912 17084 ts_tablet_manager.cc:1398] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P d4d180df6a2f4d78b0296a096b748a71: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240502 14:06:56.488847 17086 consensus_queue.cc:260] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.508102 17084 tablet_bootstrap.cc:492] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71: Bootstrap starting.
I20240502 14:06:56.509407 17086 ts_tablet_manager.cc:1398] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent starting tablet: real 0.044s user 0.007s sys 0.000s
I20240502 14:06:56.510591 17086 tablet_bootstrap.cc:492] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap starting.
I20240502 14:06:56.514083 17084 tablet_bootstrap.cc:654] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:56.517956 17085 tablet_bootstrap.cc:654] T 4b68452f8a34439f8cd577bb4be62596 P 38ec2862e51846ce84552fa0e7f025f2: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:56.524370 17086 tablet_bootstrap.cc:654] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:56.526134 17104 consensus_queue.cc:1035] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:56.539011 17085 tablet_bootstrap.cc:492] T 4b68452f8a34439f8cd577bb4be62596 P 38ec2862e51846ce84552fa0e7f025f2: No bootstrap required, opened a new log
I20240502 14:06:56.539482 17085 ts_tablet_manager.cc:1367] T 4b68452f8a34439f8cd577bb4be62596 P 38ec2862e51846ce84552fa0e7f025f2: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.001s
I20240502 14:06:56.542361 17093 consensus_queue.cc:1035] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:56.551949 17084 tablet_bootstrap.cc:492] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71: No bootstrap required, opened a new log
I20240502 14:06:56.552685 17084 ts_tablet_manager.cc:1367] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71: Time spent bootstrapping tablet: real 0.045s user 0.015s sys 0.027s
I20240502 14:06:56.556908 17085 raft_consensus.cc:348] T 4b68452f8a34439f8cd577bb4be62596 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.557966 17085 raft_consensus.cc:374] T 4b68452f8a34439f8cd577bb4be62596 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:56.558333 17085 raft_consensus.cc:729] T 4b68452f8a34439f8cd577bb4be62596 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ec2862e51846ce84552fa0e7f025f2, State: Initialized, Role: FOLLOWER
I20240502 14:06:56.559216 17085 consensus_queue.cc:260] T 4b68452f8a34439f8cd577bb4be62596 P 38ec2862e51846ce84552fa0e7f025f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.561640 17085 ts_tablet_manager.cc:1398] T 4b68452f8a34439f8cd577bb4be62596 P 38ec2862e51846ce84552fa0e7f025f2: Time spent starting tablet: real 0.022s user 0.004s sys 0.004s
I20240502 14:06:56.562681 17085 tablet_bootstrap.cc:492] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P 38ec2862e51846ce84552fa0e7f025f2: Bootstrap starting.
I20240502 14:06:56.561030 17084 raft_consensus.cc:348] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.563205 17084 raft_consensus.cc:374] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:56.563547 17084 raft_consensus.cc:729] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4d180df6a2f4d78b0296a096b748a71, State: Initialized, Role: FOLLOWER
I20240502 14:06:56.565737 17084 consensus_queue.cc:260] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.569002 17085 tablet_bootstrap.cc:654] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P 38ec2862e51846ce84552fa0e7f025f2: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:56.569579 17084 ts_tablet_manager.cc:1398] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71: Time spent starting tablet: real 0.016s user 0.007s sys 0.003s
I20240502 14:06:56.570866 17084 tablet_bootstrap.cc:492] T e5ed65313b9b45eb86098f6208983522 P d4d180df6a2f4d78b0296a096b748a71: Bootstrap starting.
I20240502 14:06:56.578256 17084 tablet_bootstrap.cc:654] T e5ed65313b9b45eb86098f6208983522 P d4d180df6a2f4d78b0296a096b748a71: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:56.585131 17085 tablet_bootstrap.cc:492] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P 38ec2862e51846ce84552fa0e7f025f2: No bootstrap required, opened a new log
I20240502 14:06:56.585773 17085 ts_tablet_manager.cc:1367] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P 38ec2862e51846ce84552fa0e7f025f2: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.010s
I20240502 14:06:56.589146 17085 raft_consensus.cc:348] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.590633 17085 raft_consensus.cc:374] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:56.590978 17085 raft_consensus.cc:729] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ec2862e51846ce84552fa0e7f025f2, State: Initialized, Role: FOLLOWER
I20240502 14:06:56.592417 17084 tablet_bootstrap.cc:492] T e5ed65313b9b45eb86098f6208983522 P d4d180df6a2f4d78b0296a096b748a71: No bootstrap required, opened a new log
I20240502 14:06:56.591745 17085 consensus_queue.cc:260] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P 38ec2862e51846ce84552fa0e7f025f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.593058 17084 ts_tablet_manager.cc:1367] T e5ed65313b9b45eb86098f6208983522 P d4d180df6a2f4d78b0296a096b748a71: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.004s
I20240502 14:06:56.595003 17085 ts_tablet_manager.cc:1398] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P 38ec2862e51846ce84552fa0e7f025f2: Time spent starting tablet: real 0.009s user 0.005s sys 0.002s
I20240502 14:06:56.595988 17084 raft_consensus.cc:348] T e5ed65313b9b45eb86098f6208983522 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.596704 17084 raft_consensus.cc:374] T e5ed65313b9b45eb86098f6208983522 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:56.596985 17084 raft_consensus.cc:729] T e5ed65313b9b45eb86098f6208983522 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4d180df6a2f4d78b0296a096b748a71, State: Initialized, Role: FOLLOWER
I20240502 14:06:56.597751 17084 consensus_queue.cc:260] T e5ed65313b9b45eb86098f6208983522 P d4d180df6a2f4d78b0296a096b748a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.599720 17084 ts_tablet_manager.cc:1398] T e5ed65313b9b45eb86098f6208983522 P d4d180df6a2f4d78b0296a096b748a71: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 14:06:56.599874 17086 tablet_bootstrap.cc:492] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d: No bootstrap required, opened a new log
I20240502 14:06:56.600348 17086 ts_tablet_manager.cc:1367] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent bootstrapping tablet: real 0.090s user 0.029s sys 0.026s
I20240502 14:06:56.602859 17086 raft_consensus.cc:348] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.603483 17086 raft_consensus.cc:374] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:56.603758 17086 raft_consensus.cc:729] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e9c2da7a3b4267b3365a1f23bfb17d, State: Initialized, Role: FOLLOWER
I20240502 14:06:56.604332 17086 consensus_queue.cc:260] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.606302 17086 ts_tablet_manager.cc:1398] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240502 14:06:56.607205 17086 tablet_bootstrap.cc:492] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap starting.
I20240502 14:06:56.611466 17086 tablet_bootstrap.cc:654] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:56.616075 17086 tablet_bootstrap.cc:492] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d: No bootstrap required, opened a new log
I20240502 14:06:56.616436 17086 ts_tablet_manager.cc:1367] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240502 14:06:56.618373 17086 raft_consensus.cc:348] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.618876 17086 raft_consensus.cc:374] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:56.619086 17086 raft_consensus.cc:729] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e9c2da7a3b4267b3365a1f23bfb17d, State: Initialized, Role: FOLLOWER
I20240502 14:06:56.619642 17086 consensus_queue.cc:260] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.621248 17086 ts_tablet_manager.cc:1398] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:06:56.621722 17091 raft_consensus.cc:482] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:56.621982 17086 tablet_bootstrap.cc:492] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap starting.
I20240502 14:06:56.622200 17091 raft_consensus.cc:504] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.624547 17091 leader_election.cc:288] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993), d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327)
I20240502 14:06:56.635231 17086 tablet_bootstrap.cc:654] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d: Neither blocks nor log segments found. Creating new log.
I20240502 14:06:56.641016 17090 raft_consensus.cc:482] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:56.641698 17090 raft_consensus.cc:504] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.647136 17090 leader_election.cc:288] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993), 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66:35709)
I20240502 14:06:56.661821 16890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68fb7e1190b949d98edd9f015142ff58" candidate_uuid: "38ec2862e51846ce84552fa0e7f025f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" is_pre_election: true
I20240502 14:06:56.662679 16890 raft_consensus.cc:2454] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38ec2862e51846ce84552fa0e7f025f2 in term 0.
I20240502 14:06:56.664809 16928 leader_election.cc:302] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ec2862e51846ce84552fa0e7f025f2, b0e9c2da7a3b4267b3365a1f23bfb17d; no voters:
I20240502 14:06:56.665702 17105 consensus_queue.cc:1035] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:06:56.665707 17091 raft_consensus.cc:2781] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:56.670569 17091 raft_consensus.cc:482] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:56.671154 17091 raft_consensus.cc:3037] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:56.670954 17039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68fb7e1190b949d98edd9f015142ff58" candidate_uuid: "38ec2862e51846ce84552fa0e7f025f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4d180df6a2f4d78b0296a096b748a71" is_pre_election: true
I20240502 14:06:56.671662 17039 raft_consensus.cc:2454] T 68fb7e1190b949d98edd9f015142ff58 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38ec2862e51846ce84552fa0e7f025f2 in term 0.
I20240502 14:06:56.674953 17086 tablet_bootstrap.cc:492] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d: No bootstrap required, opened a new log
I20240502 14:06:56.676919 17086 ts_tablet_manager.cc:1367] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent bootstrapping tablet: real 0.055s user 0.005s sys 0.036s
I20240502 14:06:56.678509 17091 raft_consensus.cc:504] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.679924 17086 raft_consensus.cc:348] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.680423 17091 leader_election.cc:288] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 1 election: Requested vote from peers b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993), d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327)
I20240502 14:06:56.680737 17086 raft_consensus.cc:374] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:06:56.681022 17086 raft_consensus.cc:729] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e9c2da7a3b4267b3365a1f23bfb17d, State: Initialized, Role: FOLLOWER
I20240502 14:06:56.681532 17038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68fb7e1190b949d98edd9f015142ff58" candidate_uuid: "38ec2862e51846ce84552fa0e7f025f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4d180df6a2f4d78b0296a096b748a71"
I20240502 14:06:56.681594 17086 consensus_queue.cc:260] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.682313 17038 raft_consensus.cc:3037] T 68fb7e1190b949d98edd9f015142ff58 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:56.683665 17086 ts_tablet_manager.cc:1398] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 14:06:56.683853 16890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68fb7e1190b949d98edd9f015142ff58" candidate_uuid: "38ec2862e51846ce84552fa0e7f025f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d"
I20240502 14:06:56.684432 16890 raft_consensus.cc:3037] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:56.688477 17093 consensus_queue.cc:1035] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:56.693599 17038 raft_consensus.cc:2454] T 68fb7e1190b949d98edd9f015142ff58 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38ec2862e51846ce84552fa0e7f025f2 in term 1.
I20240502 14:06:56.694799 16929 leader_election.cc:302] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ec2862e51846ce84552fa0e7f025f2, d4d180df6a2f4d78b0296a096b748a71; no voters:
I20240502 14:06:56.696523 17147 raft_consensus.cc:482] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:56.696732 17091 raft_consensus.cc:2781] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:56.697172 17091 raft_consensus.cc:686] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 LEADER]: Becoming Leader. State: Replica: 38ec2862e51846ce84552fa0e7f025f2, State: Running, Role: LEADER
I20240502 14:06:56.696946 17147 raft_consensus.cc:504] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.697973 17091 consensus_queue.cc:237] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.699013 17147 leader_election.cc:288] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993), d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327)
I20240502 14:06:56.699187 16889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b68452f8a34439f8cd577bb4be62596" candidate_uuid: "d4d180df6a2f4d78b0296a096b748a71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" is_pre_election: true
I20240502 14:06:56.699816 16889 raft_consensus.cc:2454] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4d180df6a2f4d78b0296a096b748a71 in term 0.
I20240502 14:06:56.699949 17038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ed65313b9b45eb86098f6208983522" candidate_uuid: "38ec2862e51846ce84552fa0e7f025f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4d180df6a2f4d78b0296a096b748a71" is_pre_election: true
I20240502 14:06:56.700462 17104 raft_consensus.cc:482] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:56.700560 17038 raft_consensus.cc:2454] T e5ed65313b9b45eb86098f6208983522 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38ec2862e51846ce84552fa0e7f025f2 in term 0.
I20240502 14:06:56.700893 17104 raft_consensus.cc:504] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.701300 17002 leader_election.cc:302] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0e9c2da7a3b4267b3365a1f23bfb17d, d4d180df6a2f4d78b0296a096b748a71; no voters:
I20240502 14:06:56.702035 17090 raft_consensus.cc:2781] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:56.702370 17090 raft_consensus.cc:482] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:56.702739 17090 raft_consensus.cc:3037] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:56.703655 16965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ce8743e49b4ef68a2ddaa7a9a55a76" candidate_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ec2862e51846ce84552fa0e7f025f2" is_pre_election: true
I20240502 14:06:56.703975 16929 leader_election.cc:302] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ec2862e51846ce84552fa0e7f025f2, d4d180df6a2f4d78b0296a096b748a71; no voters:
I20240502 14:06:56.704289 16965 raft_consensus.cc:2454] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0e9c2da7a3b4267b3365a1f23bfb17d in term 0.
I20240502 14:06:56.702735 17104 leader_election.cc:288] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66:35709), d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327)
I20240502 14:06:56.702237 16889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ed65313b9b45eb86098f6208983522" candidate_uuid: "38ec2862e51846ce84552fa0e7f025f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" is_pre_election: true
I20240502 14:06:56.705420 16889 raft_consensus.cc:2454] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38ec2862e51846ce84552fa0e7f025f2 in term 0.
I20240502 14:06:56.706121 17147 raft_consensus.cc:2781] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:56.706306 16965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b68452f8a34439f8cd577bb4be62596" candidate_uuid: "d4d180df6a2f4d78b0296a096b748a71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ec2862e51846ce84552fa0e7f025f2" is_pre_election: true
I20240502 14:06:56.706544 17147 raft_consensus.cc:482] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:56.706966 16965 raft_consensus.cc:2454] T 4b68452f8a34439f8cd577bb4be62596 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4d180df6a2f4d78b0296a096b748a71 in term 0.
I20240502 14:06:56.706986 17147 raft_consensus.cc:3037] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:56.710472 16855 leader_election.cc:302] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ec2862e51846ce84552fa0e7f025f2, b0e9c2da7a3b4267b3365a1f23bfb17d; no voters:
I20240502 14:06:56.711225 17104 raft_consensus.cc:2781] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:06:56.711577 17104 raft_consensus.cc:482] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:56.711881 17104 raft_consensus.cc:3037] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:56.711769 17090 raft_consensus.cc:504] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.715129 16889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b68452f8a34439f8cd577bb4be62596" candidate_uuid: "d4d180df6a2f4d78b0296a096b748a71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d"
I20240502 14:06:56.715344 17090 leader_election.cc:288] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71 [CANDIDATE]: Term 1 election: Requested vote from peers b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993), 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66:35709)
I20240502 14:06:56.716928 16965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b68452f8a34439f8cd577bb4be62596" candidate_uuid: "d4d180df6a2f4d78b0296a096b748a71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ec2862e51846ce84552fa0e7f025f2"
I20240502 14:06:56.717707 16965 raft_consensus.cc:3037] T 4b68452f8a34439f8cd577bb4be62596 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:56.718746 17104 raft_consensus.cc:504] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.707293 17038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ce8743e49b4ef68a2ddaa7a9a55a76" candidate_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4d180df6a2f4d78b0296a096b748a71" is_pre_election: true
I20240502 14:06:56.715672 16889 raft_consensus.cc:3037] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:56.721248 16964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ce8743e49b4ef68a2ddaa7a9a55a76" candidate_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ec2862e51846ce84552fa0e7f025f2"
I20240502 14:06:56.721557 17039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ce8743e49b4ef68a2ddaa7a9a55a76" candidate_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4d180df6a2f4d78b0296a096b748a71"
I20240502 14:06:56.722015 16964 raft_consensus.cc:3037] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P 38ec2862e51846ce84552fa0e7f025f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:56.722954 17104 leader_election.cc:288] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d [CANDIDATE]: Term 1 election: Requested vote from peers 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66:35709), d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327)
I20240502 14:06:56.725818 17147 raft_consensus.cc:504] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.726545 16964 raft_consensus.cc:2454] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0e9c2da7a3b4267b3365a1f23bfb17d in term 1.
I20240502 14:06:56.727550 16855 leader_election.cc:302] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38ec2862e51846ce84552fa0e7f025f2, b0e9c2da7a3b4267b3365a1f23bfb17d; no voters: d4d180df6a2f4d78b0296a096b748a71
I20240502 14:06:56.728174 17104 raft_consensus.cc:2781] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:56.728721 17104 raft_consensus.cc:686] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 LEADER]: Becoming Leader. State: Replica: b0e9c2da7a3b4267b3365a1f23bfb17d, State: Running, Role: LEADER
I20240502 14:06:56.721329 17038 raft_consensus.cc:2454] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0e9c2da7a3b4267b3365a1f23bfb17d in term 0.
I20240502 14:06:56.729494 16888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ed65313b9b45eb86098f6208983522" candidate_uuid: "38ec2862e51846ce84552fa0e7f025f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d"
I20240502 14:06:56.729635 17104 consensus_queue.cc:237] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.730222 16888 raft_consensus.cc:3037] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:56.733052 16965 raft_consensus.cc:2454] T 4b68452f8a34439f8cd577bb4be62596 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4d180df6a2f4d78b0296a096b748a71 in term 1.
I20240502 14:06:56.736481 16889 raft_consensus.cc:2454] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4d180df6a2f4d78b0296a096b748a71 in term 1.
I20240502 14:06:56.736923 16888 raft_consensus.cc:2454] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38ec2862e51846ce84552fa0e7f025f2 in term 1.
I20240502 14:06:56.728407 16890 raft_consensus.cc:2454] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38ec2862e51846ce84552fa0e7f025f2 in term 1.
I20240502 14:06:56.737722 17002 leader_election.cc:302] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0e9c2da7a3b4267b3365a1f23bfb17d, d4d180df6a2f4d78b0296a096b748a71; no voters:
I20240502 14:06:56.737442 16788 catalog_manager.cc:5484] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d reported cstate change: term changed from 0 to 1, leader changed from <none> to b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65). New cstate: current_term: 1 leader_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:56.738883 17090 raft_consensus.cc:2781] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:56.738729 16928 leader_election.cc:302] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ec2862e51846ce84552fa0e7f025f2, b0e9c2da7a3b4267b3365a1f23bfb17d; no voters:
I20240502 14:06:56.738782 17038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ed65313b9b45eb86098f6208983522" candidate_uuid: "38ec2862e51846ce84552fa0e7f025f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4d180df6a2f4d78b0296a096b748a71"
I20240502 14:06:56.739689 17038 raft_consensus.cc:3037] T e5ed65313b9b45eb86098f6208983522 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:06:56.739812 17091 raft_consensus.cc:2781] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:06:56.740602 17090 raft_consensus.cc:686] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71 [term 1 LEADER]: Becoming Leader. State: Replica: d4d180df6a2f4d78b0296a096b748a71, State: Running, Role: LEADER
I20240502 14:06:56.742112 17091 raft_consensus.cc:686] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 LEADER]: Becoming Leader. State: Replica: 38ec2862e51846ce84552fa0e7f025f2, State: Running, Role: LEADER
I20240502 14:06:56.742843 17091 consensus_queue.cc:237] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.742578 17090 consensus_queue.cc:237] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:56.740798 17147 leader_election.cc:288] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 1 election: Requested vote from peers b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993), d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327)
I20240502 14:06:56.751569 17038 raft_consensus.cc:2454] T e5ed65313b9b45eb86098f6208983522 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38ec2862e51846ce84552fa0e7f025f2 in term 1.
I20240502 14:06:56.756605 16788 catalog_manager.cc:5484] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67). New cstate: current_term: 1 leader_uuid: "d4d180df6a2f4d78b0296a096b748a71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } health_report { overall_health: HEALTHY } } }
I20240502 14:06:56.766384 16786 catalog_manager.cc:5484] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66). New cstate: current_term: 1 leader_uuid: "38ec2862e51846ce84552fa0e7f025f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:56.767609 16786 catalog_manager.cc:5484] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66). New cstate: current_term: 1 leader_uuid: "38ec2862e51846ce84552fa0e7f025f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:56.782517 11301 tablet_server.cc:178] TabletServer@127.11.9.65:0 shutting down...
I20240502 14:06:56.802948 11301 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 14:06:56.803725 11301 tablet_replica.cc:330] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d: stopping tablet replica
I20240502 14:06:56.804462 11301 raft_consensus.cc:2229] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:56.804796 11301 raft_consensus.cc:2258] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:56.806591 11301 tablet_replica.cc:330] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d: stopping tablet replica
I20240502 14:06:56.807060 11301 raft_consensus.cc:2229] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:56.807343 11301 raft_consensus.cc:2258] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:56.808964 11301 tablet_replica.cc:330] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d: stopping tablet replica
I20240502 14:06:56.809396 11301 raft_consensus.cc:2229] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:56.809914 11301 pending_rounds.cc:62] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d: Trying to abort 1 pending ops.
I20240502 14:06:56.810073 11301 pending_rounds.cc:68] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023242513332531200 op_type: NO_OP noop_request { }
I20240502 14:06:56.810348 11301 raft_consensus.cc:2866] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 14:06:56.810552 11301 raft_consensus.cc:2258] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:56.812523 11301 tablet_replica.cc:330] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d: stopping tablet replica
I20240502 14:06:56.812959 11301 raft_consensus.cc:2229] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:56.813241 11301 raft_consensus.cc:2258] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:56.814903 11301 tablet_replica.cc:330] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d: stopping tablet replica
I20240502 14:06:56.815339 11301 raft_consensus.cc:2229] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:56.816071 11301 raft_consensus.cc:2258] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:56.817989 11301 tablet_replica.cc:330] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d: stopping tablet replica
I20240502 14:06:56.818428 11301 raft_consensus.cc:2229] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:56.819065 11301 raft_consensus.cc:2258] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:56.820911 11301 tablet_replica.cc:330] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d: stopping tablet replica
I20240502 14:06:56.821328 11301 raft_consensus.cc:2229] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:56.821985 11301 raft_consensus.cc:2258] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:56.823638 11301 tablet_replica.cc:330] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d: stopping tablet replica
I20240502 14:06:56.824020 11301 raft_consensus.cc:2229] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:56.824684 11301 raft_consensus.cc:2258] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:56.840606 11301 tablet_server.cc:194] TabletServer@127.11.9.65:0 shutdown complete.
W20240502 14:06:57.132648 16928 consensus_peers.cc:485] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2 -> Peer b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993): Couldn't send request to peer b0e9c2da7a3b4267b3365a1f23bfb17d. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.65:44993: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:57.171099 17146 consensus_queue.cc:1035] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:06:57.183903 16928 consensus_peers.cc:485] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2 -> Peer b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993): Couldn't send request to peer b0e9c2da7a3b4267b3365a1f23bfb17d. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.65:44993: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:57.218768 17156 consensus_queue.cc:1035] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:06:57.232712 17090 consensus_queue.cc:1035] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:06:57.246047 17002 consensus_peers.cc:485] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71 -> Peer b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993): Couldn't send request to peer b0e9c2da7a3b4267b3365a1f23bfb17d. Status: Network error: Client connection negotiation failed: client connection to 127.11.9.65:44993: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:57.867008 16788 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49968:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20240502 14:06:57.867440 16788 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49968:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20240502 14:06:57.881183 16788 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 854848cec8144d36b0e06dd804df9293: Sending DeleteTablet for 3 replicas of tablet 47ce8743e49b4ef68a2ddaa7a9a55a76
I20240502 14:06:57.882319 16788 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 854848cec8144d36b0e06dd804df9293: Sending DeleteTablet for 3 replicas of tablet 68fb7e1190b949d98edd9f015142ff58
I20240502 14:06:57.883284 16953 tablet_service.cc:1514] Processing DeleteTablet for tablet 47ce8743e49b4ef68a2ddaa7a9a55a76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:06:57 UTC) from {username='slave'} at 127.0.0.1:34996
I20240502 14:06:57.884238 17029 tablet_service.cc:1514] Processing DeleteTablet for tablet 47ce8743e49b4ef68a2ddaa7a9a55a76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:06:57 UTC) from {username='slave'} at 127.0.0.1:60084
I20240502 14:06:57.884558 17028 tablet_service.cc:1514] Processing DeleteTablet for tablet 68fb7e1190b949d98edd9f015142ff58 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:06:57 UTC) from {username='slave'} at 127.0.0.1:60084
I20240502 14:06:57.884531 16788 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 854848cec8144d36b0e06dd804df9293: Sending DeleteTablet for 3 replicas of tablet 4b68452f8a34439f8cd577bb4be62596
I20240502 14:06:57.886541 16955 tablet_service.cc:1514] Processing DeleteTablet for tablet 68fb7e1190b949d98edd9f015142ff58 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:06:57 UTC) from {username='slave'} at 127.0.0.1:34996
I20240502 14:06:57.887575 16955 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b68452f8a34439f8cd577bb4be62596 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:06:57 UTC) from {username='slave'} at 127.0.0.1:34996
I20240502 14:06:57.889750 17028 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b68452f8a34439f8cd577bb4be62596 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:06:57 UTC) from {username='slave'} at 127.0.0.1:60084
I20240502 14:06:57.890457 16788 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 854848cec8144d36b0e06dd804df9293: Sending DeleteTablet for 3 replicas of tablet e5ed65313b9b45eb86098f6208983522
I20240502 14:06:57.892835 16955 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ed65313b9b45eb86098f6208983522 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:06:57 UTC) from {username='slave'} at 127.0.0.1:34996
I20240502 14:06:57.895162 17028 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ed65313b9b45eb86098f6208983522 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:06:57 UTC) from {username='slave'} at 127.0.0.1:60084
I20240502 14:06:57.898651 17171 tablet_replica.cc:330] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P d4d180df6a2f4d78b0296a096b748a71: stopping tablet replica
I20240502 14:06:57.899014 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 14:06:57.899042 17169 tablet_replica.cc:330] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P 38ec2862e51846ce84552fa0e7f025f2: stopping tablet replica
I20240502 14:06:57.899582 17171 raft_consensus.cc:2229] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:57.900091 17171 raft_consensus.cc:2258] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P d4d180df6a2f4d78b0296a096b748a71 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:57.900097 17169 raft_consensus.cc:2229] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:57.902175 17169 raft_consensus.cc:2258] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:57.902503 17171 ts_tablet_manager.cc:1875] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P d4d180df6a2f4d78b0296a096b748a71: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 14:06:57.903414 16772 catalog_manager.cc:4573] TS b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4b68452f8a34439f8cd577bb4be62596: Network error: Client connection negotiation failed: client connection to 127.11.9.65:44993: connect: Connection refused (error 111)
I20240502 14:06:57.904837 17169 ts_tablet_manager.cc:1875] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P 38ec2862e51846ce84552fa0e7f025f2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 14:06:57.907344 17172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:06:57.908341 17173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:57.915076 11301 server_base.cc:1034] running on GCE node
W20240502 14:06:57.916620 17175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:57.917668 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:06:57.917948 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:57.918156 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658817918138 us; error 0 us; skew 500 ppm
I20240502 14:06:57.918891 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:57.919523 17171 ts_tablet_manager.cc:1888] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P d4d180df6a2f4d78b0296a096b748a71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 14:06:57.919884 17171 log.cc:1196] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P d4d180df6a2f4d78b0296a096b748a71: Deleting WAL directory at /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-2-root/wals/47ce8743e49b4ef68a2ddaa7a9a55a76
I20240502 14:06:57.920732 17171 ts_tablet_manager.cc:1909] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P d4d180df6a2f4d78b0296a096b748a71: Deleting consensus metadata
I20240502 14:06:57.921924 17169 ts_tablet_manager.cc:1888] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P 38ec2862e51846ce84552fa0e7f025f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 14:06:57.922325 17169 log.cc:1196] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P 38ec2862e51846ce84552fa0e7f025f2: Deleting WAL directory at /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-1-root/wals/47ce8743e49b4ef68a2ddaa7a9a55a76
I20240502 14:06:57.922807 17169 ts_tablet_manager.cc:1909] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P 38ec2862e51846ce84552fa0e7f025f2: Deleting consensus metadata
I20240502 14:06:57.923534 11301 webserver.cc:460] Webserver started at http://127.11.9.68:33033/ using document root <none> and password file <none>
I20240502 14:06:57.924151 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:57.924008 16773 catalog_manager.cc:4830] TS d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327): tablet 47ce8743e49b4ef68a2ddaa7a9a55a76 (table dugtrio [id=319ac21d405a474db2378120a46398fb]) successfully deleted
I20240502 14:06:57.924448 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:06:57.924834 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:06:57.925305 17169 tablet_replica.cc:330] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2: stopping tablet replica
I20240502 14:06:57.925267 16774 catalog_manager.cc:4830] TS 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66:35709): tablet 47ce8743e49b4ef68a2ddaa7a9a55a76 (table dugtrio [id=319ac21d405a474db2378120a46398fb]) successfully deleted
I20240502 14:06:57.924880 17171 tablet_replica.cc:330] T 68fb7e1190b949d98edd9f015142ff58 P d4d180df6a2f4d78b0296a096b748a71: stopping tablet replica
I20240502 14:06:57.926198 17169 raft_consensus.cc:2229] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:57.926304 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-3-root/instance:
uuid: "4ea7e1236be744dfbc70ab6f087dd645"
format_stamp: "Formatted at 2024-05-02 14:06:57 on dist-test-slave-j72s"
I20240502 14:06:57.926757 17171 raft_consensus.cc:2229] T 68fb7e1190b949d98edd9f015142ff58 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:57.927053 17169 raft_consensus.cc:2258] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:57.927322 17171 raft_consensus.cc:2258] T 68fb7e1190b949d98edd9f015142ff58 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:57.929283 17169 ts_tablet_manager.cc:1875] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:06:57.930462 17171 ts_tablet_manager.cc:1875] T 68fb7e1190b949d98edd9f015142ff58 P d4d180df6a2f4d78b0296a096b748a71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:06:57.931203 17150 raft_consensus.cc:482] T dd932332efd64937b1142facd80b7be0 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b0e9c2da7a3b4267b3365a1f23bfb17d)
I20240502 14:06:57.931672 17150 raft_consensus.cc:504] T dd932332efd64937b1142facd80b7be0 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
W20240502 14:06:57.938259 17002 leader_election.cc:334] T dd932332efd64937b1142facd80b7be0 P d4d180df6a2f4d78b0296a096b748a71 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993): Network error: Client connection negotiation failed: client connection to 127.11.9.65:44993: connect: Connection refused (error 111)
I20240502 14:06:57.939162 17150 leader_election.cc:288] T dd932332efd64937b1142facd80b7be0 P d4d180df6a2f4d78b0296a096b748a71 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993), 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66:35709)
I20240502 14:06:57.940708 16963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd932332efd64937b1142facd80b7be0" candidate_uuid: "d4d180df6a2f4d78b0296a096b748a71" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "38ec2862e51846ce84552fa0e7f025f2" is_pre_election: true
I20240502 14:06:57.942396 17004 leader_election.cc:302] T dd932332efd64937b1142facd80b7be0 P d4d180df6a2f4d78b0296a096b748a71 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4d180df6a2f4d78b0296a096b748a71; no voters: 38ec2862e51846ce84552fa0e7f025f2, b0e9c2da7a3b4267b3365a1f23bfb17d
I20240502 14:06:57.942862 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.002s sys 0.009s
I20240502 14:06:57.943109 17169 ts_tablet_manager.cc:1888] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 14:06:57.943485 17169 log.cc:1196] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2: Deleting WAL directory at /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-1-root/wals/68fb7e1190b949d98edd9f015142ff58
I20240502 14:06:57.944161 17169 ts_tablet_manager.cc:1909] T 68fb7e1190b949d98edd9f015142ff58 P 38ec2862e51846ce84552fa0e7f025f2: Deleting consensus metadata
I20240502 14:06:57.946869 17169 tablet_replica.cc:330] T 4b68452f8a34439f8cd577bb4be62596 P 38ec2862e51846ce84552fa0e7f025f2: stopping tablet replica
I20240502 14:06:57.946827 16774 catalog_manager.cc:4830] TS 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66:35709): tablet 68fb7e1190b949d98edd9f015142ff58 (table dugtrio [id=319ac21d405a474db2378120a46398fb]) successfully deleted
I20240502 14:06:57.948149 17169 raft_consensus.cc:2229] T 4b68452f8a34439f8cd577bb4be62596 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:57.948657 17171 ts_tablet_manager.cc:1888] T 68fb7e1190b949d98edd9f015142ff58 P d4d180df6a2f4d78b0296a096b748a71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 14:06:57.948875 17169 raft_consensus.cc:2258] T 4b68452f8a34439f8cd577bb4be62596 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:57.949991 17150 raft_consensus.cc:2726] T dd932332efd64937b1142facd80b7be0 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:06:57.949059 17171 log.cc:1196] T 68fb7e1190b949d98edd9f015142ff58 P d4d180df6a2f4d78b0296a096b748a71: Deleting WAL directory at /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-2-root/wals/68fb7e1190b949d98edd9f015142ff58
I20240502 14:06:57.950521 17156 raft_consensus.cc:482] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b0e9c2da7a3b4267b3365a1f23bfb17d)
I20240502 14:06:57.951231 17171 ts_tablet_manager.cc:1909] T 68fb7e1190b949d98edd9f015142ff58 P d4d180df6a2f4d78b0296a096b748a71: Deleting consensus metadata
I20240502 14:06:57.951025 17156 raft_consensus.cc:504] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:57.952239 17169 ts_tablet_manager.cc:1875] T 4b68452f8a34439f8cd577bb4be62596 P 38ec2862e51846ce84552fa0e7f025f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:06:57.953044 17180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:57.953135 17156 leader_election.cc:288] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993), d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327)
I20240502 14:06:57.954305 17171 tablet_replica.cc:330] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71: stopping tablet replica
I20240502 14:06:57.954722 17038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd932332efd64937b1142facd80b7be0" candidate_uuid: "38ec2862e51846ce84552fa0e7f025f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4d180df6a2f4d78b0296a096b748a71" is_pre_election: true
I20240502 14:06:57.955739 16773 catalog_manager.cc:4830] TS d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327): tablet 68fb7e1190b949d98edd9f015142ff58 (table dugtrio [id=319ac21d405a474db2378120a46398fb]) successfully deleted
I20240502 14:06:57.956022 17038 raft_consensus.cc:2454] T dd932332efd64937b1142facd80b7be0 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38ec2862e51846ce84552fa0e7f025f2 in term 1.
I20240502 14:06:57.956579 17171 raft_consensus.cc:2229] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:57.957288 16929 leader_election.cc:302] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ec2862e51846ce84552fa0e7f025f2, d4d180df6a2f4d78b0296a096b748a71; no voters:
I20240502 14:06:57.957687 17171 raft_consensus.cc:2258] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:57.958151 17156 raft_consensus.cc:2781] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:06:57.958516 17156 raft_consensus.cc:482] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b0e9c2da7a3b4267b3365a1f23bfb17d)
I20240502 14:06:57.958918 17156 raft_consensus.cc:3037] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:06:57.959357 16928 leader_election.cc:334] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993): Network error: Client connection negotiation failed: client connection to 127.11.9.65:44993: connect: Connection refused (error 111)
I20240502 14:06:57.955060 11301 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.004s
I20240502 14:06:57.961843 17171 ts_tablet_manager.cc:1875] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:06:57.961918 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-3-root
uuid: "4ea7e1236be744dfbc70ab6f087dd645"
format_stamp: "Formatted at 2024-05-02 14:06:57 on dist-test-slave-j72s"
I20240502 14:06:57.962600 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:57.965400 17156 raft_consensus.cc:504] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:57.969184 17169 ts_tablet_manager.cc:1888] T 4b68452f8a34439f8cd577bb4be62596 P 38ec2862e51846ce84552fa0e7f025f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 14:06:57.969702 17169 log.cc:1196] T 4b68452f8a34439f8cd577bb4be62596 P 38ec2862e51846ce84552fa0e7f025f2: Deleting WAL directory at /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-1-root/wals/4b68452f8a34439f8cd577bb4be62596
I20240502 14:06:57.970506 17169 ts_tablet_manager.cc:1909] T 4b68452f8a34439f8cd577bb4be62596 P 38ec2862e51846ce84552fa0e7f025f2: Deleting consensus metadata
I20240502 14:06:57.970628 17038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd932332efd64937b1142facd80b7be0" candidate_uuid: "38ec2862e51846ce84552fa0e7f025f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4d180df6a2f4d78b0296a096b748a71"
I20240502 14:06:57.971334 17038 raft_consensus.cc:3037] T dd932332efd64937b1142facd80b7be0 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:57.973896 17169 tablet_replica.cc:330] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2: stopping tablet replica
I20240502 14:06:57.974933 17169 raft_consensus.cc:2229] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:06:57.976056 17169 raft_consensus.cc:2258] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:57.978266 16774 catalog_manager.cc:4830] TS 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66:35709): tablet 4b68452f8a34439f8cd577bb4be62596 (table dugtrio [id=319ac21d405a474db2378120a46398fb]) successfully deleted
W20240502 14:06:57.978344 16928 leader_election.cc:334] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993): Network error: Client connection negotiation failed: client connection to 127.11.9.65:44993: connect: Connection refused (error 111)
I20240502 14:06:57.981149 17171 ts_tablet_manager.cc:1888] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 14:06:57.981650 17171 log.cc:1196] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71: Deleting WAL directory at /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-2-root/wals/4b68452f8a34439f8cd577bb4be62596
I20240502 14:06:57.981684 17038 raft_consensus.cc:2454] T dd932332efd64937b1142facd80b7be0 P d4d180df6a2f4d78b0296a096b748a71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38ec2862e51846ce84552fa0e7f025f2 in term 2.
I20240502 14:06:57.982797 17171 ts_tablet_manager.cc:1909] T 4b68452f8a34439f8cd577bb4be62596 P d4d180df6a2f4d78b0296a096b748a71: Deleting consensus metadata
I20240502 14:06:57.983074 16929 leader_election.cc:302] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38ec2862e51846ce84552fa0e7f025f2, d4d180df6a2f4d78b0296a096b748a71; no voters: b0e9c2da7a3b4267b3365a1f23bfb17d
I20240502 14:06:57.983888 17169 ts_tablet_manager.cc:1875] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:06:57.986486 17171 tablet_replica.cc:330] T e5ed65313b9b45eb86098f6208983522 P d4d180df6a2f4d78b0296a096b748a71: stopping tablet replica
I20240502 14:06:57.986558 16773 catalog_manager.cc:4830] TS d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327): tablet 4b68452f8a34439f8cd577bb4be62596 (table dugtrio [id=319ac21d405a474db2378120a46398fb]) successfully deleted
I20240502 14:06:57.987910 17171 raft_consensus.cc:2229] T e5ed65313b9b45eb86098f6208983522 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:57.988423 17156 leader_election.cc:288] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 2 election: Requested vote from peers b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993), d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327)
I20240502 14:06:57.988636 17171 raft_consensus.cc:2258] T e5ed65313b9b45eb86098f6208983522 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:57.988970 17156 raft_consensus.cc:2781] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:06:57.989416 17156 raft_consensus.cc:686] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [term 2 LEADER]: Becoming Leader. State: Replica: 38ec2862e51846ce84552fa0e7f025f2, State: Running, Role: LEADER
I20240502 14:06:57.990520 17156 consensus_queue.cc:237] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:57.999858 17171 ts_tablet_manager.cc:1875] T e5ed65313b9b45eb86098f6208983522 P d4d180df6a2f4d78b0296a096b748a71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:06:57.999812 16788 catalog_manager.cc:5484] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 reported cstate change: term changed from 1 to 2, leader changed from b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65) to 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66). New cstate: current_term: 2 leader_uuid: "38ec2862e51846ce84552fa0e7f025f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:58.002154 17169 ts_tablet_manager.cc:1888] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 14:06:58.002568 17169 log.cc:1196] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2: Deleting WAL directory at /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-1-root/wals/e5ed65313b9b45eb86098f6208983522
I20240502 14:06:58.003219 17169 ts_tablet_manager.cc:1909] T e5ed65313b9b45eb86098f6208983522 P 38ec2862e51846ce84552fa0e7f025f2: Deleting consensus metadata
I20240502 14:06:58.006631 16774 catalog_manager.cc:4830] TS 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66:35709): tablet e5ed65313b9b45eb86098f6208983522 (table dugtrio [id=319ac21d405a474db2378120a46398fb]) successfully deleted
I20240502 14:06:58.010466 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:58.011581 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:58.013198 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:58.013468 17171 ts_tablet_manager.cc:1888] T e5ed65313b9b45eb86098f6208983522 P d4d180df6a2f4d78b0296a096b748a71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 14:06:58.013810 17171 log.cc:1196] T e5ed65313b9b45eb86098f6208983522 P d4d180df6a2f4d78b0296a096b748a71: Deleting WAL directory at /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-2-root/wals/e5ed65313b9b45eb86098f6208983522
I20240502 14:06:58.014317 17171 ts_tablet_manager.cc:1909] T e5ed65313b9b45eb86098f6208983522 P d4d180df6a2f4d78b0296a096b748a71: Deleting consensus metadata
I20240502 14:06:58.015950 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:06:58.016208 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:58.016469 11301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:06:58.016728 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:58.016486 16773 catalog_manager.cc:4830] TS d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327): tablet e5ed65313b9b45eb86098f6208983522 (table dugtrio [id=319ac21d405a474db2378120a46398fb]) successfully deleted
I20240502 14:06:58.031320 17150 raft_consensus.cc:482] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b0e9c2da7a3b4267b3365a1f23bfb17d)
I20240502 14:06:58.032030 17150 raft_consensus.cc:504] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:58.034945 17150 leader_election.cc:288] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993), 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66:35709)
I20240502 14:06:58.035094 17156 raft_consensus.cc:482] T f476570d1ef4476ba22a79694a340e88 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b0e9c2da7a3b4267b3365a1f23bfb17d)
I20240502 14:06:58.035787 16963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f476570d1ef4476ba22a79694a340e88" candidate_uuid: "d4d180df6a2f4d78b0296a096b748a71" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "38ec2862e51846ce84552fa0e7f025f2" is_pre_election: true
I20240502 14:06:58.035712 17156 raft_consensus.cc:504] T f476570d1ef4476ba22a79694a340e88 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:58.036716 16963 raft_consensus.cc:2454] T f476570d1ef4476ba22a79694a340e88 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4d180df6a2f4d78b0296a096b748a71 in term 1.
I20240502 14:06:58.037763 17156 leader_election.cc:288] T f476570d1ef4476ba22a79694a340e88 P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993), d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327)
I20240502 14:06:58.038652 17004 leader_election.cc:302] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ec2862e51846ce84552fa0e7f025f2, d4d180df6a2f4d78b0296a096b748a71; no voters:
I20240502 14:06:58.039537 17150 raft_consensus.cc:2781] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240502 14:06:58.039556 17002 leader_election.cc:334] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993): Network error: Client connection negotiation failed: client connection to 127.11.9.65:44993: connect: Connection refused (error 111)
I20240502 14:06:58.039983 17150 raft_consensus.cc:482] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b0e9c2da7a3b4267b3365a1f23bfb17d)
I20240502 14:06:58.040362 17150 raft_consensus.cc:3037] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:58.040390 17038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f476570d1ef4476ba22a79694a340e88" candidate_uuid: "38ec2862e51846ce84552fa0e7f025f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4d180df6a2f4d78b0296a096b748a71" is_pre_election: true
W20240502 14:06:58.043602 16928 leader_election.cc:334] T f476570d1ef4476ba22a79694a340e88 P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993): Network error: Client connection negotiation failed: client connection to 127.11.9.65:44993: connect: Connection refused (error 111)
I20240502 14:06:58.047502 17156 raft_consensus.cc:482] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b0e9c2da7a3b4267b3365a1f23bfb17d)
I20240502 14:06:58.047966 17156 raft_consensus.cc:504] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:58.049253 17150 raft_consensus.cc:504] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:58.050446 17038 raft_consensus.cc:2379] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38ec2862e51846ce84552fa0e7f025f2 in current term 2: Already voted for candidate d4d180df6a2f4d78b0296a096b748a71 in this term.
I20240502 14:06:58.053903 16929 leader_election.cc:302] T f476570d1ef4476ba22a79694a340e88 P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38ec2862e51846ce84552fa0e7f025f2; no voters: b0e9c2da7a3b4267b3365a1f23bfb17d, d4d180df6a2f4d78b0296a096b748a71
W20240502 14:06:58.054193 16928 leader_election.cc:334] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993): Network error: Client connection negotiation failed: client connection to 127.11.9.65:44993: connect: Connection refused (error 111)
I20240502 14:06:58.054968 17183 raft_consensus.cc:3037] T f476570d1ef4476ba22a79694a340e88 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:58.055152 17156 leader_election.cc:288] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993), d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327)
I20240502 14:06:58.056686 17038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94012a9173e74331942e131aeb26320c" candidate_uuid: "38ec2862e51846ce84552fa0e7f025f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4d180df6a2f4d78b0296a096b748a71" is_pre_election: true
I20240502 14:06:58.057322 17038 raft_consensus.cc:2454] T 94012a9173e74331942e131aeb26320c P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38ec2862e51846ce84552fa0e7f025f2 in term 1.
I20240502 14:06:58.060976 16929 leader_election.cc:302] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38ec2862e51846ce84552fa0e7f025f2, d4d180df6a2f4d78b0296a096b748a71; no voters: b0e9c2da7a3b4267b3365a1f23bfb17d
I20240502 14:06:58.062144 17156 raft_consensus.cc:2781] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:06:58.062518 17156 raft_consensus.cc:482] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b0e9c2da7a3b4267b3365a1f23bfb17d)
I20240502 14:06:58.062856 17156 raft_consensus.cc:3037] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:58.064980 17183 raft_consensus.cc:2726] T f476570d1ef4476ba22a79694a340e88 P 38ec2862e51846ce84552fa0e7f025f2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:06:58.067768 16963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f476570d1ef4476ba22a79694a340e88" candidate_uuid: "d4d180df6a2f4d78b0296a096b748a71" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "38ec2862e51846ce84552fa0e7f025f2"
I20240502 14:06:58.070454 17156 raft_consensus.cc:504] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:58.075059 16963 raft_consensus.cc:2454] T f476570d1ef4476ba22a79694a340e88 P 38ec2862e51846ce84552fa0e7f025f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4d180df6a2f4d78b0296a096b748a71 in term 2.
I20240502 14:06:58.076210 17004 leader_election.cc:302] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ec2862e51846ce84552fa0e7f025f2, d4d180df6a2f4d78b0296a096b748a71; no voters:
W20240502 14:06:58.076436 16928 leader_election.cc:334] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993): Network error: Client connection negotiation failed: client connection to 127.11.9.65:44993: connect: Connection refused (error 111)
I20240502 14:06:58.079043 17038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94012a9173e74331942e131aeb26320c" candidate_uuid: "38ec2862e51846ce84552fa0e7f025f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4d180df6a2f4d78b0296a096b748a71"
I20240502 14:06:58.079759 17038 raft_consensus.cc:3037] T 94012a9173e74331942e131aeb26320c P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:58.081992 17150 leader_election.cc:288] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [CANDIDATE]: Term 2 election: Requested vote from peers b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993), 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66:35709)
I20240502 14:06:58.087815 17038 raft_consensus.cc:2454] T 94012a9173e74331942e131aeb26320c P d4d180df6a2f4d78b0296a096b748a71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38ec2862e51846ce84552fa0e7f025f2 in term 2.
I20240502 14:06:58.088408 17150 raft_consensus.cc:482] T 94012a9173e74331942e131aeb26320c P d4d180df6a2f4d78b0296a096b748a71 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:58.088938 17150 raft_consensus.cc:504] T 94012a9173e74331942e131aeb26320c P d4d180df6a2f4d78b0296a096b748a71 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:58.093495 17156 leader_election.cc:288] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 2 election: Requested vote from peers b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993), d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327)
I20240502 14:06:58.093739 17212 raft_consensus.cc:2781] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:06:58.094842 16929 leader_election.cc:302] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38ec2862e51846ce84552fa0e7f025f2, d4d180df6a2f4d78b0296a096b748a71; no voters: b0e9c2da7a3b4267b3365a1f23bfb17d
I20240502 14:06:58.095916 17156 raft_consensus.cc:2781] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:06:58.097105 17156 raft_consensus.cc:686] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2 [term 2 LEADER]: Becoming Leader. State: Replica: 38ec2862e51846ce84552fa0e7f025f2, State: Running, Role: LEADER
I20240502 14:06:58.097674 16963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94012a9173e74331942e131aeb26320c" candidate_uuid: "d4d180df6a2f4d78b0296a096b748a71" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "38ec2862e51846ce84552fa0e7f025f2" is_pre_election: true
I20240502 14:06:58.097891 17156 consensus_queue.cc:237] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:58.094220 17212 raft_consensus.cc:686] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [term 2 LEADER]: Becoming Leader. State: Replica: d4d180df6a2f4d78b0296a096b748a71, State: Running, Role: LEADER
I20240502 14:06:58.101111 17212 consensus_queue.cc:237] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:58.103418 17150 leader_election.cc:288] T 94012a9173e74331942e131aeb26320c P d4d180df6a2f4d78b0296a096b748a71 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993), 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66:35709)
W20240502 14:06:58.113138 17002 leader_election.cc:334] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993): Network error: Client connection negotiation failed: client connection to 127.11.9.65:44993: connect: Connection refused (error 111)
W20240502 14:06:58.114118 17002 leader_election.cc:334] T 94012a9173e74331942e131aeb26320c P d4d180df6a2f4d78b0296a096b748a71 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993): Network error: Client connection negotiation failed: client connection to 127.11.9.65:44993: connect: Connection refused (error 111)
I20240502 14:06:58.114655 17002 leader_election.cc:302] T 94012a9173e74331942e131aeb26320c P d4d180df6a2f4d78b0296a096b748a71 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4d180df6a2f4d78b0296a096b748a71; no voters: 38ec2862e51846ce84552fa0e7f025f2, b0e9c2da7a3b4267b3365a1f23bfb17d
I20240502 14:06:58.115618 17211 raft_consensus.cc:2726] T 94012a9173e74331942e131aeb26320c P d4d180df6a2f4d78b0296a096b748a71 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:06:58.121135 16788 catalog_manager.cc:5484] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2 reported cstate change: term changed from 1 to 2, leader changed from b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65) to 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66). New cstate: current_term: 2 leader_uuid: "38ec2862e51846ce84552fa0e7f025f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:58.123492 16786 catalog_manager.cc:5484] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 reported cstate change: term changed from 1 to 2, leader changed from b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65) to d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67). New cstate: current_term: 2 leader_uuid: "d4d180df6a2f4d78b0296a096b748a71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } health_report { overall_health: HEALTHY } } }
I20240502 14:06:58.147153 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.68:40153
I20240502 14:06:58.147261 17248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.68:40153 every 8 connection(s)
I20240502 14:06:58.151440 11301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:06:58.157826 17252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:58.165918 17249 heartbeater.cc:347] Connected to a master server at 127.11.9.126:36895
W20240502 14:06:58.166082 17253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:58.166388 17249 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:58.166932 11301 server_base.cc:1034] running on GCE node
I20240502 14:06:58.167313 17249 heartbeater.cc:511] Master 127.11.9.126:36895 requested a full tablet report, sending...
W20240502 14:06:58.167574 17256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:06:58.168572 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 14:06:58.168874 11301 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 14:06:58.169118 11301 hybrid_clock.cc:648] HybridClock initialized: now 1714658818169100 us; error 0 us; skew 500 ppm
I20240502 14:06:58.169575 16786 ts_manager.cc:194] Registered new tserver with Master: 4ea7e1236be744dfbc70ab6f087dd645 (127.11.9.68:40153)
I20240502 14:06:58.169693 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:06:58.171192 16786 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50006
I20240502 14:06:58.172565 11301 webserver.cc:460] Webserver started at http://127.11.9.65:33675/ using document root <none> and password file <none>
I20240502 14:06:58.173218 11301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:06:58.173463 11301 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240502 14:06:58.178108 11301 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 14:06:58.182258 17260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:06:58.182967 11301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20240502 14:06:58.183291 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-0-root
uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d"
format_stamp: "Formatted at 2024-05-02 14:06:54 on dist-test-slave-j72s"
I20240502 14:06:58.183543 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:06:58.187626 17212 raft_consensus.cc:482] T c6742043c1e74f269ea157de517c5e98 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b0e9c2da7a3b4267b3365a1f23bfb17d)
I20240502 14:06:58.188063 17212 raft_consensus.cc:504] T c6742043c1e74f269ea157de517c5e98 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } } peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } }
I20240502 14:06:58.189613 17212 leader_election.cc:288] T c6742043c1e74f269ea157de517c5e98 P d4d180df6a2f4d78b0296a096b748a71 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66:35709), b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993)
I20240502 14:06:58.190331 16963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6742043c1e74f269ea157de517c5e98" candidate_uuid: "d4d180df6a2f4d78b0296a096b748a71" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "38ec2862e51846ce84552fa0e7f025f2" is_pre_election: true
W20240502 14:06:58.192749 17002 leader_election.cc:334] T c6742043c1e74f269ea157de517c5e98 P d4d180df6a2f4d78b0296a096b748a71 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993): Network error: Client connection negotiation failed: client connection to 127.11.9.65:44993: connect: Connection refused (error 111)
I20240502 14:06:58.193112 17002 leader_election.cc:302] T c6742043c1e74f269ea157de517c5e98 P d4d180df6a2f4d78b0296a096b748a71 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4d180df6a2f4d78b0296a096b748a71; no voters: 38ec2862e51846ce84552fa0e7f025f2, b0e9c2da7a3b4267b3365a1f23bfb17d
I20240502 14:06:58.193830 17212 raft_consensus.cc:2726] T c6742043c1e74f269ea157de517c5e98 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:06:58.195150 11301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:06:58.196151 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:06:58.197432 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:06:58.200815 17267 ts_tablet_manager.cc:522] Loading tablet metadata (0/8 complete)
I20240502 14:06:58.235010 11301 ts_tablet_manager.cc:559] Loaded tablet metadata (8 total tablets, 8 live tablets)
I20240502 14:06:58.235306 11301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.035s user 0.002s sys 0.000s
I20240502 14:06:58.235541 11301 ts_tablet_manager.cc:574] Registering tablets (0/8 complete)
I20240502 14:06:58.239543 17267 tablet_bootstrap.cc:492] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap starting.
I20240502 14:06:58.248917 17267 tablet_bootstrap.cc:492] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:58.249655 17267 tablet_bootstrap.cc:492] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap complete.
I20240502 14:06:58.250236 17267 ts_tablet_manager.cc:1367] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240502 14:06:58.252619 17267 raft_consensus.cc:348] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:58.253190 17267 raft_consensus.cc:729] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e9c2da7a3b4267b3365a1f23bfb17d, State: Initialized, Role: FOLLOWER
I20240502 14:06:58.253765 17267 consensus_queue.cc:260] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:58.254245 17156 raft_consensus.cc:482] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b0e9c2da7a3b4267b3365a1f23bfb17d)
I20240502 14:06:58.254791 17156 raft_consensus.cc:504] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } } peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } }
I20240502 14:06:58.256327 17267 ts_tablet_manager.cc:1398] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 14:06:58.256728 17156 leader_election.cc:288] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327), b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993)
I20240502 14:06:58.257159 17267 tablet_bootstrap.cc:492] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap starting.
I20240502 14:06:58.257475 17038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6742043c1e74f269ea157de517c5e98" candidate_uuid: "38ec2862e51846ce84552fa0e7f025f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4d180df6a2f4d78b0296a096b748a71" is_pre_election: true
I20240502 14:06:58.258138 17038 raft_consensus.cc:2454] T c6742043c1e74f269ea157de517c5e98 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38ec2862e51846ce84552fa0e7f025f2 in term 1.
I20240502 14:06:58.259263 16929 leader_election.cc:302] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ec2862e51846ce84552fa0e7f025f2, d4d180df6a2f4d78b0296a096b748a71; no voters:
I20240502 14:06:58.260597 17156 raft_consensus.cc:2781] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:06:58.260941 17156 raft_consensus.cc:482] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b0e9c2da7a3b4267b3365a1f23bfb17d)
I20240502 14:06:58.261283 17156 raft_consensus.cc:3037] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:06:58.262822 16928 leader_election.cc:334] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993): Network error: Client connection negotiation failed: client connection to 127.11.9.65:44993: connect: Connection refused (error 111)
I20240502 14:06:58.268427 17156 raft_consensus.cc:504] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } } peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } }
I20240502 14:06:58.269904 11301 ts_tablet_manager.cc:590] Registered 8 tablets
I20240502 14:06:58.270179 11301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.035s user 0.034s sys 0.000s
I20240502 14:06:58.270630 17267 tablet_bootstrap.cc:492] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:58.271582 17267 tablet_bootstrap.cc:492] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap complete.
I20240502 14:06:58.271445 17038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6742043c1e74f269ea157de517c5e98" candidate_uuid: "38ec2862e51846ce84552fa0e7f025f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4d180df6a2f4d78b0296a096b748a71"
I20240502 14:06:58.272080 17038 raft_consensus.cc:3037] T c6742043c1e74f269ea157de517c5e98 P d4d180df6a2f4d78b0296a096b748a71 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:58.273247 17156 leader_election.cc:288] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 2 election: Requested vote from peers d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327), b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993)
I20240502 14:06:58.272190 17267 ts_tablet_manager.cc:1367] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20240502 14:06:58.276053 17267 raft_consensus.cc:348] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } } peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } }
I20240502 14:06:58.276773 17267 raft_consensus.cc:729] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e9c2da7a3b4267b3365a1f23bfb17d, State: Initialized, Role: FOLLOWER
I20240502 14:06:58.277474 17267 consensus_queue.cc:260] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } } peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } }
I20240502 14:06:58.279327 17038 raft_consensus.cc:2454] T c6742043c1e74f269ea157de517c5e98 P d4d180df6a2f4d78b0296a096b748a71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38ec2862e51846ce84552fa0e7f025f2 in term 2.
I20240502 14:06:58.279639 17267 ts_tablet_manager.cc:1398] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 14:06:58.280624 17267 tablet_bootstrap.cc:492] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap starting.
I20240502 14:06:58.280792 16929 leader_election.cc:302] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ec2862e51846ce84552fa0e7f025f2, d4d180df6a2f4d78b0296a096b748a71; no voters:
I20240502 14:06:58.281898 17156 raft_consensus.cc:2781] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:06:58.282500 17156 raft_consensus.cc:686] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [term 2 LEADER]: Becoming Leader. State: Replica: 38ec2862e51846ce84552fa0e7f025f2, State: Running, Role: LEADER
W20240502 14:06:58.283460 16836 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver b0e9c2da7a3b4267b3365a1f23bfb17d not available for placement
W20240502 14:06:58.283941 16928 leader_election.cc:334] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993): Network error: Client connection negotiation failed: client connection to 127.11.9.65:44993: connect: Connection refused (error 111)
I20240502 14:06:58.283262 17156 consensus_queue.cc:237] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } } peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } }
I20240502 14:06:58.295861 17267 tablet_bootstrap.cc:492] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:58.296689 17267 tablet_bootstrap.cc:492] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap complete.
I20240502 14:06:58.297341 17267 ts_tablet_manager.cc:1367] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20240502 14:06:58.298560 16786 catalog_manager.cc:5484] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 reported cstate change: term changed from 1 to 2, leader changed from b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65) to 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66). New cstate: current_term: 2 leader_uuid: "38ec2862e51846ce84552fa0e7f025f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } health_report { overall_health: UNKNOWN } } }
I20240502 14:06:58.299671 17267 raft_consensus.cc:348] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:58.300480 17267 raft_consensus.cc:729] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e9c2da7a3b4267b3365a1f23bfb17d, State: Initialized, Role: FOLLOWER
I20240502 14:06:58.301421 17267 consensus_queue.cc:260] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:58.304225 17267 ts_tablet_manager.cc:1398] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 14:06:58.305195 17267 tablet_bootstrap.cc:492] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap starting.
I20240502 14:06:58.321415 17267 tablet_bootstrap.cc:492] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:58.322078 17267 tablet_bootstrap.cc:492] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap complete.
I20240502 14:06:58.322517 17267 ts_tablet_manager.cc:1367] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20240502 14:06:58.323827 17267 raft_consensus.cc:348] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:58.324465 17267 raft_consensus.cc:729] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e9c2da7a3b4267b3365a1f23bfb17d, State: Initialized, Role: FOLLOWER
I20240502 14:06:58.325006 17267 consensus_queue.cc:260] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:58.326718 17267 ts_tablet_manager.cc:1398] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 14:06:58.327383 17267 tablet_bootstrap.cc:492] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap starting.
I20240502 14:06:58.336355 17267 tablet_bootstrap.cc:492] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:58.336948 17267 tablet_bootstrap.cc:492] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap complete.
I20240502 14:06:58.337376 17267 ts_tablet_manager.cc:1367] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20240502 14:06:58.338737 17267 raft_consensus.cc:348] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:58.339211 17267 raft_consensus.cc:729] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e9c2da7a3b4267b3365a1f23bfb17d, State: Initialized, Role: FOLLOWER
I20240502 14:06:58.339685 17267 consensus_queue.cc:260] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:58.340945 17267 ts_tablet_manager.cc:1398] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:06:58.341178 11301 rpc_server.cc:307] RPC server started. Bound to: 127.11.9.65:44993
I20240502 14:06:58.341288 17330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.9.65:44993 every 8 connection(s)
I20240502 14:06:58.341509 17267 tablet_bootstrap.cc:492] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap starting.
I20240502 14:06:58.352676 17267 tablet_bootstrap.cc:492] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:06:58.353339 17267 tablet_bootstrap.cc:492] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap complete.
I20240502 14:06:58.353964 17267 ts_tablet_manager.cc:1367] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.006s
I20240502 14:06:58.356083 17267 raft_consensus.cc:348] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:58.356869 17267 raft_consensus.cc:729] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e9c2da7a3b4267b3365a1f23bfb17d, State: Initialized, Role: FOLLOWER
I20240502 14:06:58.357398 17267 consensus_queue.cc:260] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:58.358650 17331 heartbeater.cc:347] Connected to a master server at 127.11.9.126:36895
I20240502 14:06:58.358933 17267 ts_tablet_manager.cc:1398] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20240502 14:06:58.358995 17331 heartbeater.cc:464] Registering TS with master...
I20240502 14:06:58.359602 17267 tablet_bootstrap.cc:492] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap starting.
I20240502 14:06:58.359797 17331 heartbeater.cc:511] Master 127.11.9.126:36895 requested a full tablet report, sending...
I20240502 14:06:58.364178 16786 ts_manager.cc:194] Re-registered known tserver with Master: b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993)
I20240502 14:06:58.368585 17267 tablet_bootstrap.cc:492] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:58.369129 17267 tablet_bootstrap.cc:492] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap complete.
I20240502 14:06:58.369632 17267 ts_tablet_manager.cc:1367] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent bootstrapping tablet: real 0.010s user 0.010s sys 0.000s
I20240502 14:06:58.370996 17267 raft_consensus.cc:348] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:58.371445 17267 raft_consensus.cc:729] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e9c2da7a3b4267b3365a1f23bfb17d, State: Initialized, Role: FOLLOWER
I20240502 14:06:58.371860 16786 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50014
I20240502 14:06:58.371901 17267 consensus_queue.cc:260] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:58.373683 17267 ts_tablet_manager.cc:1398] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 14:06:58.374171 17267 tablet_bootstrap.cc:492] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap starting.
I20240502 14:06:58.374451 17331 heartbeater.cc:503] Master 127.11.9.126:36895 was elected leader, sending a full tablet report...
I20240502 14:06:58.384349 17267 tablet_bootstrap.cc:492] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:58.385061 17267 tablet_bootstrap.cc:492] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d: Bootstrap complete.
I20240502 14:06:58.385571 17267 ts_tablet_manager.cc:1367] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent bootstrapping tablet: real 0.011s user 0.011s sys 0.000s
I20240502 14:06:58.387038 17267 raft_consensus.cc:348] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:58.387483 17267 raft_consensus.cc:729] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e9c2da7a3b4267b3365a1f23bfb17d, State: Initialized, Role: FOLLOWER
I20240502 14:06:58.387982 17267 consensus_queue.cc:260] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } }
I20240502 14:06:58.389324 17267 ts_tablet_manager.cc:1398] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20240502 14:06:58.428495 17038 raft_consensus.cc:1261] T dd932332efd64937b1142facd80b7be0 P d4d180df6a2f4d78b0296a096b748a71 [term 2 FOLLOWER]: Refusing update from remote peer 38ec2862e51846ce84552fa0e7f025f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:06:58.429711 17156 consensus_queue.cc:1035] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:58.449369 17305 raft_consensus.cc:3037] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:58.454623 17305 raft_consensus.cc:1261] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 2 FOLLOWER]: Refusing update from remote peer 38ec2862e51846ce84552fa0e7f025f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:06:58.455736 17183 consensus_queue.cc:1035] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:58.557253 17294 tablet_service.cc:1514] Processing DeleteTablet for tablet 47ce8743e49b4ef68a2ddaa7a9a55a76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:06:57 UTC) from {username='slave'} at 127.0.0.1:53126
I20240502 14:06:58.558598 17344 tablet_replica.cc:330] stopping tablet replica
I20240502 14:06:58.559221 17344 raft_consensus.cc:2229] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:58.559559 17344 pending_rounds.cc:62] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d: Trying to abort 1 pending ops.
I20240502 14:06:58.559715 17344 pending_rounds.cc:68] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023242513332531200 op_type: NO_OP noop_request { }
I20240502 14:06:58.560015 17344 raft_consensus.cc:2866] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240502 14:06:58.560238 17344 raft_consensus.cc:2258] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:58.562108 17344 ts_tablet_manager.cc:1875] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:06:58.570771 17038 raft_consensus.cc:1261] T 94012a9173e74331942e131aeb26320c P d4d180df6a2f4d78b0296a096b748a71 [term 2 FOLLOWER]: Refusing update from remote peer 38ec2862e51846ce84552fa0e7f025f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:06:58.571846 17183 consensus_queue.cc:1035] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:58.572180 17344 ts_tablet_manager.cc:1888] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 14:06:58.572558 17344 log.cc:1196] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d: Deleting WAL directory at /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-0-root/wals/47ce8743e49b4ef68a2ddaa7a9a55a76
I20240502 14:06:58.573145 17344 ts_tablet_manager.cc:1909] T 47ce8743e49b4ef68a2ddaa7a9a55a76 P b0e9c2da7a3b4267b3365a1f23bfb17d: Deleting consensus metadata
I20240502 14:06:58.575506 16772 catalog_manager.cc:4830] TS b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993): tablet 47ce8743e49b4ef68a2ddaa7a9a55a76 (table dugtrio [id=319ac21d405a474db2378120a46398fb]) successfully deleted
I20240502 14:06:58.580528 17294 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b68452f8a34439f8cd577bb4be62596 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:06:57 UTC) from {username='slave'} at 127.0.0.1:53126
I20240502 14:06:58.580754 17305 raft_consensus.cc:3037] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:58.581243 17344 tablet_replica.cc:330] stopping tablet replica
I20240502 14:06:58.582213 17344 raft_consensus.cc:2229] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:58.582705 17344 raft_consensus.cc:2258] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:58.585506 17344 ts_tablet_manager.cc:1875] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:06:58.587765 17305 raft_consensus.cc:1261] T 94012a9173e74331942e131aeb26320c P b0e9c2da7a3b4267b3365a1f23bfb17d [term 2 FOLLOWER]: Refusing update from remote peer 38ec2862e51846ce84552fa0e7f025f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:06:58.588968 17156 consensus_queue.cc:1035] T 94012a9173e74331942e131aeb26320c P 38ec2862e51846ce84552fa0e7f025f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:58.589572 17294 tablet_service.cc:1514] Processing DeleteTablet for tablet 68fb7e1190b949d98edd9f015142ff58 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:06:57 UTC) from {username='slave'} at 127.0.0.1:53126
I20240502 14:06:58.596848 17344 ts_tablet_manager.cc:1888] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 14:06:58.597147 17344 log.cc:1196] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d: Deleting WAL directory at /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-0-root/wals/4b68452f8a34439f8cd577bb4be62596
I20240502 14:06:58.597777 17344 ts_tablet_manager.cc:1909] T 4b68452f8a34439f8cd577bb4be62596 P b0e9c2da7a3b4267b3365a1f23bfb17d: Deleting consensus metadata
I20240502 14:06:58.600210 17344 tablet_replica.cc:330] stopping tablet replica
I20240502 14:06:58.600404 16772 catalog_manager.cc:4830] TS b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993): tablet 4b68452f8a34439f8cd577bb4be62596 (table dugtrio [id=319ac21d405a474db2378120a46398fb]) successfully deleted
I20240502 14:06:58.602021 17344 raft_consensus.cc:2229] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:58.602545 17344 raft_consensus.cc:2258] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:58.604709 17344 ts_tablet_manager.cc:1875] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:06:58.613366 17344 ts_tablet_manager.cc:1888] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 14:06:58.613651 17344 log.cc:1196] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d: Deleting WAL directory at /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-0-root/wals/68fb7e1190b949d98edd9f015142ff58
I20240502 14:06:58.614099 17344 ts_tablet_manager.cc:1909] T 68fb7e1190b949d98edd9f015142ff58 P b0e9c2da7a3b4267b3365a1f23bfb17d: Deleting consensus metadata
I20240502 14:06:58.615988 16772 catalog_manager.cc:4830] TS b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993): tablet 68fb7e1190b949d98edd9f015142ff58 (table dugtrio [id=319ac21d405a474db2378120a46398fb]) successfully deleted
I20240502 14:06:58.624599 16963 raft_consensus.cc:1261] T f476570d1ef4476ba22a79694a340e88 P 38ec2862e51846ce84552fa0e7f025f2 [term 2 FOLLOWER]: Refusing update from remote peer d4d180df6a2f4d78b0296a096b748a71: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:06:58.625015 17294 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ed65313b9b45eb86098f6208983522 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:06:57 UTC) from {username='slave'} at 127.0.0.1:53126
I20240502 14:06:58.625855 17344 tablet_replica.cc:330] stopping tablet replica
I20240502 14:06:58.626113 17212 consensus_queue.cc:1035] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:58.626662 17344 raft_consensus.cc:2229] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:58.627197 17344 raft_consensus.cc:2258] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:58.629832 17344 ts_tablet_manager.cc:1875] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:06:58.635143 17305 raft_consensus.cc:3037] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:58.642992 17305 raft_consensus.cc:1261] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 2 FOLLOWER]: Refusing update from remote peer d4d180df6a2f4d78b0296a096b748a71: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:06:58.645061 17350 consensus_queue.cc:1035] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 14:06:58.646589 17344 ts_tablet_manager.cc:1888] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 14:06:58.647012 17344 log.cc:1196] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d: Deleting WAL directory at /tmp/dist-test-taskGnDDsC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714658659912502-11301-0/minicluster-data/ts-0-root/wals/e5ed65313b9b45eb86098f6208983522
I20240502 14:06:58.647657 17344 ts_tablet_manager.cc:1909] T e5ed65313b9b45eb86098f6208983522 P b0e9c2da7a3b4267b3365a1f23bfb17d: Deleting consensus metadata
I20240502 14:06:58.650712 16772 catalog_manager.cc:4830] TS b0e9c2da7a3b4267b3365a1f23bfb17d (127.11.9.65:44993): tablet e5ed65313b9b45eb86098f6208983522 (table dugtrio [id=319ac21d405a474db2378120a46398fb]) successfully deleted
I20240502 14:06:58.807794 17038 raft_consensus.cc:1261] T c6742043c1e74f269ea157de517c5e98 P d4d180df6a2f4d78b0296a096b748a71 [term 2 FOLLOWER]: Refusing update from remote peer 38ec2862e51846ce84552fa0e7f025f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:06:58.809016 17156 consensus_queue.cc:1035] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:58.816282 17305 raft_consensus.cc:3037] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:58.822686 17305 raft_consensus.cc:1261] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 2 FOLLOWER]: Refusing update from remote peer 38ec2862e51846ce84552fa0e7f025f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:06:58.823661 17183 consensus_queue.cc:1035] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:06:59.173894 17249 heartbeater.cc:503] Master 127.11.9.126:36895 was elected leader, sending a full tablet report...
W20240502 14:06:59.292001 16836 auto_rebalancer.cc:249] could not retrieve auto-rebalancing replica moves: Not found: table 7c460605d9c1462d9bb93b606e63b679: could not find any suitable replica to move from server 38ec2862e51846ce84552fa0e7f025f2 to server 4ea7e1236be744dfbc70ab6f087dd645
I20240502 14:07:00.311863 16963 consensus_queue.cc:237] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } } peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } attrs { replace: true } } peers { permanent_uuid: "4ea7e1236be744dfbc70ab6f087dd645" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 40153 } attrs { promote: true } }
I20240502 14:07:00.319576 17038 raft_consensus.cc:1261] T c6742043c1e74f269ea157de517c5e98 P d4d180df6a2f4d78b0296a096b748a71 [term 2 FOLLOWER]: Refusing update from remote peer 38ec2862e51846ce84552fa0e7f025f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:07:00.320251 17304 raft_consensus.cc:1261] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 2 FOLLOWER]: Refusing update from remote peer 38ec2862e51846ce84552fa0e7f025f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:07:00.321089 17357 consensus_queue.cc:1035] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:07:00.322115 17156 consensus_queue.cc:1035] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 14:07:00.328644 16928 consensus_peers.cc:485] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 -> Peer 4ea7e1236be744dfbc70ab6f087dd645 (127.11.9.68:40153): Couldn't send request to peer 4ea7e1236be744dfbc70ab6f087dd645. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6742043c1e74f269ea157de517c5e98. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:07:00.329964 17357 raft_consensus.cc:2932] T c6742043c1e74f269ea157de517c5e98 P 38ec2862e51846ce84552fa0e7f025f2 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4ea7e1236be744dfbc70ab6f087dd645 (127.11.9.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } } peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } attrs { replace: true } } peers { permanent_uuid: "4ea7e1236be744dfbc70ab6f087dd645" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 40153 } attrs { promote: true } } }
I20240502 14:07:00.332286 17038 raft_consensus.cc:2932] T c6742043c1e74f269ea157de517c5e98 P d4d180df6a2f4d78b0296a096b748a71 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4ea7e1236be744dfbc70ab6f087dd645 (127.11.9.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } } peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } attrs { replace: true } } peers { permanent_uuid: "4ea7e1236be744dfbc70ab6f087dd645" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 40153 } attrs { promote: true } } }
I20240502 14:07:00.333362 17304 raft_consensus.cc:2932] T c6742043c1e74f269ea157de517c5e98 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4ea7e1236be744dfbc70ab6f087dd645 (127.11.9.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } } peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } attrs { replace: true } } peers { permanent_uuid: "4ea7e1236be744dfbc70ab6f087dd645" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 40153 } attrs { promote: true } } }
I20240502 14:07:00.344605 16786 catalog_manager.cc:5484] T c6742043c1e74f269ea157de517c5e98 P d4d180df6a2f4d78b0296a096b748a71 reported cstate change: config changed from index -1 to 3, NON_VOTER 4ea7e1236be744dfbc70ab6f087dd645 (127.11.9.68) added. New cstate: current_term: 2 leader_uuid: "38ec2862e51846ce84552fa0e7f025f2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } } peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } attrs { replace: true } } peers { permanent_uuid: "4ea7e1236be744dfbc70ab6f087dd645" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 40153 } attrs { promote: true } } }
I20240502 14:07:00.357995 17038 consensus_queue.cc:237] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } attrs { replace: true } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } } peers { permanent_uuid: "4ea7e1236be744dfbc70ab6f087dd645" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 40153 } attrs { promote: true } }
I20240502 14:07:00.368232 16963 raft_consensus.cc:1261] T f476570d1ef4476ba22a79694a340e88 P 38ec2862e51846ce84552fa0e7f025f2 [term 2 FOLLOWER]: Refusing update from remote peer d4d180df6a2f4d78b0296a096b748a71: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:07:00.368865 17304 raft_consensus.cc:1261] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 2 FOLLOWER]: Refusing update from remote peer d4d180df6a2f4d78b0296a096b748a71: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:07:00.370640 17212 consensus_queue.cc:1035] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:07:00.371510 17370 consensus_queue.cc:1035] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 14:07:00.378686 17002 consensus_peers.cc:485] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 -> Peer 4ea7e1236be744dfbc70ab6f087dd645 (127.11.9.68:40153): Couldn't send request to peer 4ea7e1236be744dfbc70ab6f087dd645. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f476570d1ef4476ba22a79694a340e88. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:07:00.381254 17370 raft_consensus.cc:2932] T f476570d1ef4476ba22a79694a340e88 P d4d180df6a2f4d78b0296a096b748a71 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4ea7e1236be744dfbc70ab6f087dd645 (127.11.9.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } attrs { replace: true } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } } peers { permanent_uuid: "4ea7e1236be744dfbc70ab6f087dd645" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 40153 } attrs { promote: true } } }
I20240502 14:07:00.382318 17304 raft_consensus.cc:2932] T f476570d1ef4476ba22a79694a340e88 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4ea7e1236be744dfbc70ab6f087dd645 (127.11.9.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } attrs { replace: true } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } } peers { permanent_uuid: "4ea7e1236be744dfbc70ab6f087dd645" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 40153 } attrs { promote: true } } }
I20240502 14:07:00.382879 16963 raft_consensus.cc:2932] T f476570d1ef4476ba22a79694a340e88 P 38ec2862e51846ce84552fa0e7f025f2 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4ea7e1236be744dfbc70ab6f087dd645 (127.11.9.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } attrs { replace: true } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } } peers { permanent_uuid: "4ea7e1236be744dfbc70ab6f087dd645" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 40153 } attrs { promote: true } } }
I20240502 14:07:00.392920 16788 catalog_manager.cc:5484] T f476570d1ef4476ba22a79694a340e88 P 38ec2862e51846ce84552fa0e7f025f2 reported cstate change: config changed from index -1 to 3, NON_VOTER 4ea7e1236be744dfbc70ab6f087dd645 (127.11.9.68) added. New cstate: current_term: 2 leader_uuid: "d4d180df6a2f4d78b0296a096b748a71" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } attrs { replace: true } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } } peers { permanent_uuid: "4ea7e1236be744dfbc70ab6f087dd645" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 40153 } attrs { promote: true } } }
I20240502 14:07:00.398686 16963 consensus_queue.cc:237] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } attrs { replace: true } } peers { permanent_uuid: "4ea7e1236be744dfbc70ab6f087dd645" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 40153 } attrs { promote: true } }
I20240502 14:07:00.405874 17038 raft_consensus.cc:1261] T dd932332efd64937b1142facd80b7be0 P d4d180df6a2f4d78b0296a096b748a71 [term 2 FOLLOWER]: Refusing update from remote peer 38ec2862e51846ce84552fa0e7f025f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:07:00.412287 17357 consensus_queue.cc:1035] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 14:07:00.416139 16928 consensus_peers.cc:485] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 -> Peer 4ea7e1236be744dfbc70ab6f087dd645 (127.11.9.68:40153): Couldn't send request to peer 4ea7e1236be744dfbc70ab6f087dd645. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd932332efd64937b1142facd80b7be0. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:07:00.418004 17304 raft_consensus.cc:1261] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 2 FOLLOWER]: Refusing update from remote peer 38ec2862e51846ce84552fa0e7f025f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:07:00.419209 17156 consensus_queue.cc:1035] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:07:00.420516 17362 raft_consensus.cc:2932] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4ea7e1236be744dfbc70ab6f087dd645 (127.11.9.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } attrs { replace: true } } peers { permanent_uuid: "4ea7e1236be744dfbc70ab6f087dd645" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 40153 } attrs { promote: true } } }
I20240502 14:07:00.422086 17038 raft_consensus.cc:2932] T dd932332efd64937b1142facd80b7be0 P d4d180df6a2f4d78b0296a096b748a71 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4ea7e1236be744dfbc70ab6f087dd645 (127.11.9.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } attrs { replace: true } } peers { permanent_uuid: "4ea7e1236be744dfbc70ab6f087dd645" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 40153 } attrs { promote: true } } }
I20240502 14:07:00.431669 17304 raft_consensus.cc:2932] T dd932332efd64937b1142facd80b7be0 P b0e9c2da7a3b4267b3365a1f23bfb17d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4ea7e1236be744dfbc70ab6f087dd645 (127.11.9.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } attrs { replace: true } } peers { permanent_uuid: "4ea7e1236be744dfbc70ab6f087dd645" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 40153 } attrs { promote: true } } }
I20240502 14:07:00.435344 16786 catalog_manager.cc:5484] T dd932332efd64937b1142facd80b7be0 P 38ec2862e51846ce84552fa0e7f025f2 reported cstate change: config changed from index -1 to 3, NON_VOTER 4ea7e1236be744dfbc70ab6f087dd645 (127.11.9.68) added. New cstate: current_term: 2 leader_uuid: "38ec2862e51846ce84552fa0e7f025f2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0e9c2da7a3b4267b3365a1f23bfb17d" member_type: VOTER last_known_addr { host: "127.11.9.65" port: 44993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38ec2862e51846ce84552fa0e7f025f2" member_type: VOTER last_known_addr { host: "127.11.9.66" port: 35709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4d180df6a2f4d78b0296a096b748a71" member_type: VOTER last_known_addr { host: "127.11.9.67" port: 35327 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ea7e1236be744dfbc70ab6f087dd645" member_type: NON_VOTER last_known_addr { host: "127.11.9.68" port: 40153 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:07:00.875641 17378 ts_tablet_manager.cc:902] T f476570d1ef4476ba22a79694a340e88 P 4ea7e1236be744dfbc70ab6f087dd645: Initiating tablet copy from peer d4d180df6a2f4d78b0296a096b748a71 (127.11.9.67:35327)
F20240502 14:07:00.876572 17378 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-05373a49f2a34ab9a96426e17bb2508c.queue_time_us, tablet-download-pool-05373a49f2a34ab9a96426e17bb2508c.queue_time_us) (tablet-download-pool-05373a49f2a34ab9a96426e17bb2508c.run_cpu_time_us, tablet-download-pool-05373a49f2a34ab9a96426e17bb2508c.run_cpu_time_us) (tablet-download-pool-05373a49f2a34ab9a96426e17bb2508c.run_wall_time_us, tablet-download-pool-05373a49f2a34ab9a96426e17bb2508c.run_wall_time_us) (tablet-download-pool-3f82d6f517dc44f18f6d054cd4590124.queue_time_us, tablet-download-pool-3f82d6f517dc44f18f6d054cd4590124.queue_time_us) (tablet-download-pool-3f82d6f517dc44f18f6d054cd4590124.run_cpu_time_us, tablet-download-pool-3f82d6f517dc44f18f6d054cd4590124.run_cpu_time_us) (tablet-download-pool-3f82d6f517dc44f18f6d054cd4590124.run_wall_time_us, tablet-download-pool-3f82d6f517dc44f18f6d054cd4590124.run_wall_time_us) (tablet-download-pool-4b48c7659b094e068c680f9b27fbcdee.queue_time_us, tablet-download-pool-4b48c7659b094e068c680f9b27fbcdee.queue_time_us) (tablet-download-pool-4b48c7659b094e068c680f9b27fbcdee.run_cpu_time_us, tablet-download-pool-4b48c7659b094e068c680f9b27fbcdee.run_cpu_time_us) (tablet-download-pool-4b48c7659b094e068c680f9b27fbcdee.run_wall_time_us, tablet-download-pool-4b48c7659b094e068c680f9b27fbcdee.run_wall_time_us) (tablet-download-pool-5509e46f7fc94f32b0f59c7063e87546.queue_time_us, tablet-download-pool-5509e46f7fc94f32b0f59c7063e87546.queue_time_us) (tablet-download-pool-5509e46f7fc94f32b0f59c7063e87546.run_cpu_time_us, tablet-download-pool-5509e46f7fc94f32b0f59c7063e87546.run_cpu_time_us) (tablet-download-pool-5509e46f7fc94f32b0f59c7063e87546.run_wall_time_us, tablet-download-pool-5509e46f7fc94f32b0f59c7063e87546.run_wall_time_us) (tablet-download-pool-56ac0dbb1eef484bba6413e33b6729b3.queue_time_us, tablet-download-pool-56ac0dbb1eef484bba6413e33b6729b3.queue_time_us) (tablet-download-pool-56ac0dbb1eef484bba6413e33b6729b3.run_cpu_time_us, tablet-download-pool-56ac0dbb1eef484bba6413e33b6729b3.run_cpu_time_us) (tablet-download-pool-56ac0dbb1eef484bba6413e33b6729b3.run_wall_time_us, tablet-download-pool-56ac0dbb1eef484bba6413e33b6729b3.run_wall_time_us) (tablet-download-pool-7284f9efdc5745c1a8f63af14c24d838.queue_time_us, tablet-download-pool-7284f9efdc5745c1a8f63af14c24d838.queue_time_us) (tablet-download-pool-7284f9efdc5745c1a8f63af14c24d838.run_cpu_time_us, tablet-download-pool-7284f9efdc5745c1a8f63af14c24d838.run_cpu_time_us) (tablet-download-pool-7284f9efdc5745c1a8f63af14c24d838.run_wall_time_us, tablet-download-pool-7284f9efdc5745c1a8f63af14c24d838.run_wall_time_us) (tablet-download-pool-8a727b7623ba4587bb183d2ce121ceed.queue_time_us, tablet-download-pool-8a727b7623ba4587bb183d2ce121ceed.queue_time_us) (tablet-download-pool-8a727b7623ba4587bb183d2ce121ceed.run_cpu_time_us, tablet-download-pool-8a727b7623ba4587bb183d2ce121ceed.run_cpu_time_us) (tablet-download-pool-8a727b7623ba4587bb183d2ce121ceed.run_wall_time_us, tablet-download-pool-8a727b7623ba4587bb183d2ce121ceed.run_wall_time_us) (tablet-download-pool-a96b8cef09b04ac987749a0c749499f3.queue_time_us, tablet-download-pool-a96b8cef09b04ac987749a0c749499f3.queue_time_us) (tablet-download-pool-a96b8cef09b04ac987749a0c749499f3.run_cpu_time_us, tablet-download-pool-a96b8cef09b04ac987749a0c749499f3.run_cpu_time_us) (tablet-download-pool-a96b8cef09b04ac987749a0c749499f3.run_wall_time_us, tablet-download-pool-a96b8cef09b04ac987749a0c749499f3.run_wall_time_us) (tablet-download-pool-b0d765fb4e1e49c2b654e9917e89c631.queue_time_us, tablet-download-pool-b0d765fb4e1e49c2b654e9917e89c631.queue_time_us) (tablet-download-pool-b0d765fb4e1e49c2b654e9917e89c631.run_cpu_time_us, tablet-download-pool-b0d765fb4e1e49c2b654e9917e89c631.run_cpu_time_us) (tablet-download-pool-b0d765fb4e1e49c2b654e9917e89c631.run_wall_time_us, tablet-download-pool-b0d765fb4e1e49c2b654e9917e89c631.run_wall_time_us) (tablet-download-pool-c808defc7a8b4ecda4490809db5f983b.queue_time_us, tablet-download-pool-c808defc7a8b4ecda4490809db5f983b.queue_time_us) (tablet-download-pool-c808defc7a8b4ecda4490809db5f983b.run_cpu_time_us, tablet-download-pool-c808defc7a8b4ecda4490809db5f983b.run_cpu_time_us) (tablet-download-pool-c808defc7a8b4ecda4490809db5f983b.run_wall_time_us, tablet-download-pool-c808defc7a8b4ecda4490809db5f983b.run_wall_time_us) (tablet-download-pool-dd54f15c1b87413c8541c924088c1cde.queue_time_us, tablet-download-pool-dd54f15c1b87413c8541c924088c1cde.queue_time_us) (tablet-download-pool-dd54f15c1b87413c8541c924088c1cde.run_cpu_time_us, tablet-download-pool-dd54f15c1b87413c8541c924088c1cde.run_cpu_time_us) (tablet-download-pool-dd54f15c1b87413c8541c924088c1cde.run_wall_time_us, tablet-download-pool-dd54f15c1b87413c8541c924088c1cde.run_wall_time_us) (tablet-download-pool-e2a87ab15cd7497abd4b275a559d7bba.queue_time_us, tablet-download-pool-e2a87ab15cd7497abd4b275a559d7bba.queue_time_us) (tablet-download-pool-e2a87ab15cd7497abd4b275a559d7bba.run_cpu_time_us, tablet-download-pool-e2a87ab15cd7497abd4b275a559d7bba.run_cpu_time_us) (tablet-download-pool-e2a87ab15cd7497abd4b275a559d7bba.run_wall_time_us, tablet-download-pool-e2a87ab15cd7497abd4b275a559d7bba.run_wall_time_us) (tablet-download-pool-f476570d1ef4476ba22a79694a340e88.queue_time_us, tablet-download-pool-f476570d1ef4476ba22a79694a340e88.queue_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714658820 (unix time) try "date -d @1714658820" if you are using GNU date ***
I20240502 14:07:00.907189 17379 ts_tablet_manager.cc:902] T dd932332efd64937b1142facd80b7be0 P 4ea7e1236be744dfbc70ab6f087dd645: Initiating tablet copy from peer 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66:35709)
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800002c25) received by PID 11301 (TID 0x7ff9199dd700) from PID 11301; stack trace: ***
@ 0x2e41a0 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2e6683 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
I20240502 14:07:00.936743 17381 ts_tablet_manager.cc:902] T c6742043c1e74f269ea157de517c5e98 P 4ea7e1236be744dfbc70ab6f087dd645: Initiating tablet copy from peer 38ec2862e51846ce84552fa0e7f025f2 (127.11.9.66:35709)
@ 0x7ff9d51a7980 (unknown) at ??:0
@ 0x7ff9ce71bfb7 gsignal at ??:0
@ 0x7ff9ce71d921 abort at ??:0
@ 0x2ea196 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7ff9d1511da4 google::LogMessage::SendToLog() at ??:0
@ 0x7ff9d1512950 google::LogMessage::Flush() at ??:0
@ 0x7ff9d1517a8b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff9d1e9f3c7 kudu::TraceMetrics::InternName() at ??:0
@ 0x7ff9d1e7594f kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7ff9d1e730a5 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7ff9d9c93422 kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7ff9d9c99c89 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7ff9d9d2f4ef kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7ff9d9d603cf kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7ff9d9d461ce kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7ff9d9d4617a _ZNSt3__18__invokeIRZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBENS_8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEE4$_12JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSH_DpOSI_ at ??:0
@ 0x7ff9d9d4610a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7ff9d9d460d2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7ff9d9d452fe std::__1::__function::__func<>::operator()() at ??:0
@ 0x7ff9db0ebda5 std::__1::__function::__value_func<>::operator()() at ??:0
@ 0x7ff9db0ebbda std::__1::function<>::operator()() at ??:0
@ 0x7ff9d1e77237 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff9d1e7a692 kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff9d1e7a64a _ZNSt3__18__invokeIRZN4kudu10ThreadPool12CreateThreadEvE3$_2JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS5_DpOS6_ at ??:0
@ 0x7ff9d1e7a5da std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7ff9d1e7a5a2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7ff9d1e7989e std::__1::__function::__func<>::operator()() at ??:0
@ 0x7ff9db0ebda5 std::__1::__function::__value_func<>::operator()() at ??:0
@ 0x7ff9db0ebbda std::__1::function<>::operator()() at ??:0
@ 0x7ff9d1e59937 kudu::Thread::SuperviseThread() at ??:0