[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 19:59:36.264292 24854 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.69.190:44235,127.24.69.189:38259,127.24.69.188:33459
I20240501 19:59:36.265779 24854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:59:36.266692 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:59:36.284730 24860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:36.284984 24861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:36.285895 24863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:36.286298 24854 server_base.cc:1034] running on GCE node
I20240501 19:59:37.404498 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:59:37.404731 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:59:37.404883 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593577404868 us; error 0 us; skew 500 ppm
I20240501 19:59:37.405639 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 19:59:37.410912 24854 webserver.cc:460] Webserver started at http://127.24.69.190:35287/ using document root <none> and password file <none>
I20240501 19:59:37.411671 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 19:59:37.411873 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:59:37.412283 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:59:37.418677 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-0-root/instance:
uuid: "933994e2d10e49ec870bcce9517ef12f"
format_stamp: "Formatted at 2024-05-01 19:59:37 on dist-test-slave-z8x3"
I20240501 19:59:37.426545 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20240501 19:59:37.431403 24869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:37.432489 24854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:59:37.432813 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-0-root
uuid: "933994e2d10e49ec870bcce9517ef12f"
format_stamp: "Formatted at 2024-05-01 19:59:37 on dist-test-slave-z8x3"
I20240501 19:59:37.433135 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:59:37.488780 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:59:37.490195 24854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:59:37.490631 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:59:37.567399 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.190:44235
I20240501 19:59:37.567461 24920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.190:44235 every 8 connection(s)
I20240501 19:59:37.573385 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 19:59:37.574987 24921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 19:59:37.581661 24923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:37.582470 24924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:37.587994 24926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:37.588728 24854 server_base.cc:1034] running on GCE node
I20240501 19:59:37.589821 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:59:37.590025 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:59:37.590175 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593577590159 us; error 0 us; skew 500 ppm
I20240501 19:59:37.590688 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 19:59:37.595284 24854 webserver.cc:460] Webserver started at http://127.24.69.189:46367/ using document root <none> and password file <none>
I20240501 19:59:37.595773 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 19:59:37.595963 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:59:37.596210 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:59:37.597272 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-1-root/instance:
uuid: "60e8576e4d1d4ceaa4483102024efe5e"
format_stamp: "Formatted at 2024-05-01 19:59:37 on dist-test-slave-z8x3"
I20240501 19:59:37.598652 24921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.69.190" port: 44235 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:59:37.603410 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20240501 19:59:37.609690 24933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:37.610508 24854 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20240501 19:59:37.610765 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-1-root
uuid: "60e8576e4d1d4ceaa4483102024efe5e"
format_stamp: "Formatted at 2024-05-01 19:59:37 on dist-test-slave-z8x3"
I20240501 19:59:37.611058 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:59:37.624692 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:59:37.626305 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:59:37.628719 24921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.69.189" port: 38259 } has no permanent_uuid. Determining permanent_uuid...
W20240501 19:59:37.630954 24873 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.69.189:38259: connect: Connection refused (error 111)
W20240501 19:59:37.635828 24921 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.24.69.189:38259: Network error: Client connection negotiation failed: client connection to 127.24.69.189:38259: connect: Connection refused (error 111)
I20240501 19:59:37.685333 24921 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.69.189" port: 38259 } attempt: 1
W20240501 19:59:37.690976 24921 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.24.69.189:38259: Network error: Client connection negotiation failed: client connection to 127.24.69.189:38259: connect: Connection refused (error 111)
I20240501 19:59:37.704057 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.189:38259
I20240501 19:59:37.704231 24985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.189:38259 every 8 connection(s)
I20240501 19:59:37.708758 24986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:37.709564 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:59:37.716852 24988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:37.717283 24986 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.69.190" port: 44235 } has no permanent_uuid. Determining permanent_uuid...
W20240501 19:59:37.720772 24989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:37.725850 24854 server_base.cc:1034] running on GCE node
W20240501 19:59:37.726390 24991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:37.727239 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:59:37.727420 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:59:37.727586 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593577727569 us; error 0 us; skew 500 ppm
I20240501 19:59:37.728322 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 19:59:37.731823 24854 webserver.cc:460] Webserver started at http://127.24.69.188:42163/ using document root <none> and password file <none>
I20240501 19:59:37.732406 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 19:59:37.732614 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:59:37.732939 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:59:37.732926 24986 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.69.189" port: 38259 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:59:37.734815 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-2-root/instance:
uuid: "298e8ee11034451391a1c69810f14cce"
format_stamp: "Formatted at 2024-05-01 19:59:37 on dist-test-slave-z8x3"
I20240501 19:59:37.740510 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 19:59:37.745261 24998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:37.745672 24986 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.69.188" port: 33459 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:59:37.746086 24854 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20240501 19:59:37.746357 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-2-root
uuid: "298e8ee11034451391a1c69810f14cce"
format_stamp: "Formatted at 2024-05-01 19:59:37 on dist-test-slave-z8x3"
I20240501 19:59:37.746647 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240501 19:59:37.750623 24986 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.24.69.188:33459: Network error: Client connection negotiation failed: client connection to 127.24.69.188:33459: connect: Connection refused (error 111)
I20240501 19:59:37.759341 24921 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.69.189" port: 38259 } attempt: 2
I20240501 19:59:37.768054 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:59:37.768496 24921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.69.188" port: 33459 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:59:37.769398 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
W20240501 19:59:37.772639 24921 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.24.69.188:33459: Network error: Client connection negotiation failed: client connection to 127.24.69.188:33459: connect: Connection refused (error 111)
I20240501 19:59:37.794041 24986 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.69.188" port: 33459 } attempt: 1
W20240501 19:59:37.799331 24986 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.24.69.188:33459: Network error: Client connection negotiation failed: client connection to 127.24.69.188:33459: connect: Connection refused (error 111)
I20240501 19:59:37.804124 24921 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.69.188" port: 33459 } attempt: 1
W20240501 19:59:37.807996 24921 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.24.69.188:33459: Network error: Client connection negotiation failed: client connection to 127.24.69.188:33459: connect: Connection refused (error 111)
I20240501 19:59:37.843218 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.188:33459
I20240501 19:59:37.843302 25050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.188:33459 every 8 connection(s)
I20240501 19:59:37.847558 24854 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 19:59:37.848413 25051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:37.853288 25051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.69.190" port: 44235 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:59:37.864151 25051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.69.189" port: 38259 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:59:37.872385 25051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.69.188" port: 33459 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:59:37.875046 24986 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.69.188" port: 33459 } attempt: 2
I20240501 19:59:37.875468 24921 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.69.188" port: 33459 } attempt: 2
I20240501 19:59:37.894804 25051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce: Bootstrap starting.
I20240501 19:59:37.898733 24986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60e8576e4d1d4ceaa4483102024efe5e: Bootstrap starting.
I20240501 19:59:37.899207 24921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 933994e2d10e49ec870bcce9517ef12f: Bootstrap starting.
I20240501 19:59:37.900624 25051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:37.902884 25051 log.cc:824] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce: Log is configured to *not* fsync() on all Append() calls
I20240501 19:59:37.905957 24921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 933994e2d10e49ec870bcce9517ef12f: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:37.906335 24986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60e8576e4d1d4ceaa4483102024efe5e: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:37.909340 25051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce: No bootstrap required, opened a new log
I20240501 19:59:37.915074 24986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60e8576e4d1d4ceaa4483102024efe5e: No bootstrap required, opened a new log
I20240501 19:59:37.915299 24921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 933994e2d10e49ec870bcce9517ef12f: No bootstrap required, opened a new log
I20240501 19:59:37.930991 25051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933994e2d10e49ec870bcce9517ef12f" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 44235 } } peers { permanent_uuid: "60e8576e4d1d4ceaa4483102024efe5e" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 38259 } } peers { permanent_uuid: "298e8ee11034451391a1c69810f14cce" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 33459 } }
I20240501 19:59:37.930917 24921 raft_consensus.cc:348] T 00000000000000000000000000000000 P 933994e2d10e49ec870bcce9517ef12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933994e2d10e49ec870bcce9517ef12f" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 44235 } } peers { permanent_uuid: "60e8576e4d1d4ceaa4483102024efe5e" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 38259 } } peers { permanent_uuid: "298e8ee11034451391a1c69810f14cce" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 33459 } }
I20240501 19:59:37.930917 24986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60e8576e4d1d4ceaa4483102024efe5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933994e2d10e49ec870bcce9517ef12f" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 44235 } } peers { permanent_uuid: "60e8576e4d1d4ceaa4483102024efe5e" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 38259 } } peers { permanent_uuid: "298e8ee11034451391a1c69810f14cce" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 33459 } }
I20240501 19:59:37.931751 25051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:37.931886 24921 raft_consensus.cc:374] T 00000000000000000000000000000000 P 933994e2d10e49ec870bcce9517ef12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:37.931959 24986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60e8576e4d1d4ceaa4483102024efe5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:37.932134 25051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 298e8ee11034451391a1c69810f14cce, State: Initialized, Role: FOLLOWER
I20240501 19:59:37.932339 24921 raft_consensus.cc:729] T 00000000000000000000000000000000 P 933994e2d10e49ec870bcce9517ef12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 933994e2d10e49ec870bcce9517ef12f, State: Initialized, Role: FOLLOWER
I20240501 19:59:37.932366 24986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60e8576e4d1d4ceaa4483102024efe5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e8576e4d1d4ceaa4483102024efe5e, State: Initialized, Role: FOLLOWER
I20240501 19:59:37.933506 25051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933994e2d10e49ec870bcce9517ef12f" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 44235 } } peers { permanent_uuid: "60e8576e4d1d4ceaa4483102024efe5e" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 38259 } } peers { permanent_uuid: "298e8ee11034451391a1c69810f14cce" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 33459 } }
I20240501 19:59:37.933531 24921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 933994e2d10e49ec870bcce9517ef12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933994e2d10e49ec870bcce9517ef12f" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 44235 } } peers { permanent_uuid: "60e8576e4d1d4ceaa4483102024efe5e" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 38259 } } peers { permanent_uuid: "298e8ee11034451391a1c69810f14cce" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 33459 } }
I20240501 19:59:37.933707 24986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60e8576e4d1d4ceaa4483102024efe5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933994e2d10e49ec870bcce9517ef12f" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 44235 } } peers { permanent_uuid: "60e8576e4d1d4ceaa4483102024efe5e" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 38259 } } peers { permanent_uuid: "298e8ee11034451391a1c69810f14cce" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 33459 } }
I20240501 19:59:37.938248 25051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:59:37.937836 25062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 933994e2d10e49ec870bcce9517ef12f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933994e2d10e49ec870bcce9517ef12f" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 44235 } } peers { permanent_uuid: "60e8576e4d1d4ceaa4483102024efe5e" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 38259 } } peers { permanent_uuid: "298e8ee11034451391a1c69810f14cce" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 33459 } } }
I20240501 19:59:37.938903 25062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 933994e2d10e49ec870bcce9517ef12f [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:59:37.939265 24921 sys_catalog.cc:564] T 00000000000000000000000000000000 P 933994e2d10e49ec870bcce9517ef12f [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:59:37.941531 25061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60e8576e4d1d4ceaa4483102024efe5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933994e2d10e49ec870bcce9517ef12f" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 44235 } } peers { permanent_uuid: "60e8576e4d1d4ceaa4483102024efe5e" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 38259 } } peers { permanent_uuid: "298e8ee11034451391a1c69810f14cce" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 33459 } } }
I20240501 19:59:37.942749 25061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60e8576e4d1d4ceaa4483102024efe5e [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:59:37.942855 24986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60e8576e4d1d4ceaa4483102024efe5e [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:59:37.943819 25060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933994e2d10e49ec870bcce9517ef12f" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 44235 } } peers { permanent_uuid: "60e8576e4d1d4ceaa4483102024efe5e" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 38259 } } peers { permanent_uuid: "298e8ee11034451391a1c69810f14cce" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 33459 } } }
I20240501 19:59:37.944959 25060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:59:37.988157 24854 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20240501 19:59:37.996044 24875 tablet.cc:2369] T 00000000000000000000000000000000 P 933994e2d10e49ec870bcce9517ef12f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:59:38.008114 25090 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 933994e2d10e49ec870bcce9517ef12f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:59:38.008114 25095 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:59:38.008597 25090 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240501 19:59:38.008814 25095 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 19:59:38.009460 24854 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240501 19:59:38.011426 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:59:38.014544 25096 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60e8576e4d1d4ceaa4483102024efe5e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:59:38.014863 25096 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240501 19:59:38.021997 25097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:38.025406 25098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:38.027521 25100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:38.028563 24854 server_base.cc:1034] running on GCE node
I20240501 19:59:38.029326 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:59:38.029505 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:59:38.029670 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593578029652 us; error 0 us; skew 500 ppm
I20240501 19:59:38.030138 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 19:59:38.033046 24854 webserver.cc:460] Webserver started at http://127.24.69.129:40825/ using document root <none> and password file <none>
I20240501 19:59:38.033493 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 19:59:38.033674 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:59:38.033905 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:59:38.034909 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-0-root/instance:
uuid: "73bfb5057eaa4f7f96d6a5f91931add1"
format_stamp: "Formatted at 2024-05-01 19:59:38 on dist-test-slave-z8x3"
I20240501 19:59:38.039440 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 19:59:38.042878 25105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:38.043751 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:59:38.044152 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-0-root
uuid: "73bfb5057eaa4f7f96d6a5f91931add1"
format_stamp: "Formatted at 2024-05-01 19:59:38 on dist-test-slave-z8x3"
I20240501 19:59:38.044514 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:59:38.063520 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:59:38.064976 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:59:38.068013 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:59:38.072825 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:59:38.073133 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:38.073369 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:59:38.073515 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:38.188359 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.129:34261
I20240501 19:59:38.188441 25167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.129:34261 every 8 connection(s)
I20240501 19:59:38.215294 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 19:59:38.217615 25060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:59:38.218251 25060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933994e2d10e49ec870bcce9517ef12f" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 44235 } } peers { permanent_uuid: "60e8576e4d1d4ceaa4483102024efe5e" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 38259 } } peers { permanent_uuid: "298e8ee11034451391a1c69810f14cce" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 33459 } }
I20240501 19:59:38.222141 25060 leader_election.cc:288] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 933994e2d10e49ec870bcce9517ef12f (127.24.69.190:44235), 60e8576e4d1d4ceaa4483102024efe5e (127.24.69.189:38259)
I20240501 19:59:38.222682 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "298e8ee11034451391a1c69810f14cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "933994e2d10e49ec870bcce9517ef12f" is_pre_election: true
I20240501 19:59:38.223242 24961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "298e8ee11034451391a1c69810f14cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60e8576e4d1d4ceaa4483102024efe5e" is_pre_election: true
I20240501 19:59:38.224763 24896 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 933994e2d10e49ec870bcce9517ef12f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 298e8ee11034451391a1c69810f14cce in term 0.
I20240501 19:59:38.226469 25000 leader_election.cc:302] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 298e8ee11034451391a1c69810f14cce, 933994e2d10e49ec870bcce9517ef12f; no voters:
I20240501 19:59:38.228365 25168 heartbeater.cc:347] Connected to a master server at 127.24.69.188:33459
I20240501 19:59:38.229285 25168 heartbeater.cc:464] Registering TS with master...
I20240501 19:59:38.226639 24961 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60e8576e4d1d4ceaa4483102024efe5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 298e8ee11034451391a1c69810f14cce in term 0.
I20240501 19:59:38.230573 25060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:59:38.231330 25060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:59:38.232239 25060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:38.232679 25168 heartbeater.cc:511] Master 127.24.69.188:33459 requested a full tablet report, sending...
I20240501 19:59:38.249320 25169 heartbeater.cc:347] Connected to a master server at 127.24.69.189:38259
I20240501 19:59:38.249742 25169 heartbeater.cc:464] Registering TS with master...
I20240501 19:59:38.250669 25169 heartbeater.cc:511] Master 127.24.69.189:38259 requested a full tablet report, sending...
I20240501 19:59:38.253443 25060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933994e2d10e49ec870bcce9517ef12f" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 44235 } } peers { permanent_uuid: "60e8576e4d1d4ceaa4483102024efe5e" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 38259 } } peers { permanent_uuid: "298e8ee11034451391a1c69810f14cce" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 33459 } }
I20240501 19:59:38.253975 24951 ts_manager.cc:194] Registered new tserver with Master: 73bfb5057eaa4f7f96d6a5f91931add1 (127.24.69.129:34261)
W20240501 19:59:38.262008 25176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:38.261425 25175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:38.264118 25060 leader_election.cc:288] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [CANDIDATE]: Term 1 election: Requested vote from peers 933994e2d10e49ec870bcce9517ef12f (127.24.69.190:44235), 60e8576e4d1d4ceaa4483102024efe5e (127.24.69.189:38259)
I20240501 19:59:38.266219 24961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "298e8ee11034451391a1c69810f14cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60e8576e4d1d4ceaa4483102024efe5e"
I20240501 19:59:38.266803 24961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60e8576e4d1d4ceaa4483102024efe5e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:38.266884 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "298e8ee11034451391a1c69810f14cce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "933994e2d10e49ec870bcce9517ef12f"
I20240501 19:59:38.267464 24896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 933994e2d10e49ec870bcce9517ef12f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:38.270495 25170 heartbeater.cc:347] Connected to a master server at 127.24.69.190:44235
I20240501 19:59:38.270943 25170 heartbeater.cc:464] Registering TS with master...
I20240501 19:59:38.274674 25170 heartbeater.cc:511] Master 127.24.69.190:44235 requested a full tablet report, sending...
I20240501 19:59:38.276237 25016 ts_manager.cc:194] Registered new tserver with Master: 73bfb5057eaa4f7f96d6a5f91931add1 (127.24.69.129:34261)
I20240501 19:59:38.282439 24961 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60e8576e4d1d4ceaa4483102024efe5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 298e8ee11034451391a1c69810f14cce in term 1.
I20240501 19:59:38.283617 25003 leader_election.cc:302] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 298e8ee11034451391a1c69810f14cce, 60e8576e4d1d4ceaa4483102024efe5e; no voters:
I20240501 19:59:38.284128 24896 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 933994e2d10e49ec870bcce9517ef12f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 298e8ee11034451391a1c69810f14cce in term 1.
I20240501 19:59:38.284583 25060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:59:38.291677 24886 ts_manager.cc:194] Registered new tserver with Master: 73bfb5057eaa4f7f96d6a5f91931add1 (127.24.69.129:34261)
I20240501 19:59:38.293642 25060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [term 1 LEADER]: Becoming Leader. State: Replica: 298e8ee11034451391a1c69810f14cce, State: Running, Role: LEADER
I20240501 19:59:38.294561 25060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933994e2d10e49ec870bcce9517ef12f" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 44235 } } peers { permanent_uuid: "60e8576e4d1d4ceaa4483102024efe5e" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 38259 } } peers { permanent_uuid: "298e8ee11034451391a1c69810f14cce" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 33459 } }
W20240501 19:59:38.296919 25178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:38.301028 24854 server_base.cc:1034] running on GCE node
I20240501 19:59:38.301296 25180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 298e8ee11034451391a1c69810f14cce. Latest consensus state: current_term: 1 leader_uuid: "298e8ee11034451391a1c69810f14cce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933994e2d10e49ec870bcce9517ef12f" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 44235 } } peers { permanent_uuid: "60e8576e4d1d4ceaa4483102024efe5e" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 38259 } } peers { permanent_uuid: "298e8ee11034451391a1c69810f14cce" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 33459 } } }
I20240501 19:59:38.302219 25180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [sys.catalog]: This master's current role is: LEADER
I20240501 19:59:38.302314 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:59:38.302978 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:59:38.303212 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593578303195 us; error 0 us; skew 500 ppm
I20240501 19:59:38.303725 25182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:59:38.303975 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 19:59:38.307507 24854 webserver.cc:460] Webserver started at http://127.24.69.130:40617/ using document root <none> and password file <none>
I20240501 19:59:38.308102 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 19:59:38.308367 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:59:38.308689 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:59:38.309468 25182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:59:38.310035 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-1-root/instance:
uuid: "12a4ee47452f4a958b6c5eab7223845e"
format_stamp: "Formatted at 2024-05-01 19:59:38 on dist-test-slave-z8x3"
I20240501 19:59:38.316058 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20240501 19:59:38.321209 25187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:38.323743 24854 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s
I20240501 19:59:38.324090 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-1-root
uuid: "12a4ee47452f4a958b6c5eab7223845e"
format_stamp: "Formatted at 2024-05-01 19:59:38 on dist-test-slave-z8x3"
I20240501 19:59:38.324512 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:59:38.324934 24961 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60e8576e4d1d4ceaa4483102024efe5e [term 1 FOLLOWER]: Refusing update from remote peer 298e8ee11034451391a1c69810f14cce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:59:38.326079 24896 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 933994e2d10e49ec870bcce9517ef12f [term 1 FOLLOWER]: Refusing update from remote peer 298e8ee11034451391a1c69810f14cce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:59:38.327801 25180 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8576e4d1d4ceaa4483102024efe5e" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 38259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 19:59:38.329746 25180 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [LEADER]: Connected to new peer: Peer: permanent_uuid: "933994e2d10e49ec870bcce9517ef12f" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 44235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:59:38.351394 25061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60e8576e4d1d4ceaa4483102024efe5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 298e8ee11034451391a1c69810f14cce. Latest consensus state: current_term: 1 leader_uuid: "298e8ee11034451391a1c69810f14cce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933994e2d10e49ec870bcce9517ef12f" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 44235 } } peers { permanent_uuid: "60e8576e4d1d4ceaa4483102024efe5e" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 38259 } } peers { permanent_uuid: "298e8ee11034451391a1c69810f14cce" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 33459 } } }
I20240501 19:59:38.352195 25061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60e8576e4d1d4ceaa4483102024efe5e [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:59:38.360109 25186 mvcc.cc:205] Tried to move back new op lower bound from 7022975296794562560 to 7022975296702382080. Current Snapshot: MvccSnapshot[applied={T|T < 7022975296794562560}]
I20240501 19:59:38.361686 25062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 933994e2d10e49ec870bcce9517ef12f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 298e8ee11034451391a1c69810f14cce. Latest consensus state: current_term: 1 leader_uuid: "298e8ee11034451391a1c69810f14cce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933994e2d10e49ec870bcce9517ef12f" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 44235 } } peers { permanent_uuid: "60e8576e4d1d4ceaa4483102024efe5e" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 38259 } } peers { permanent_uuid: "298e8ee11034451391a1c69810f14cce" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 33459 } } }
I20240501 19:59:38.363062 25062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 933994e2d10e49ec870bcce9517ef12f [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:59:38.371320 25060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "298e8ee11034451391a1c69810f14cce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933994e2d10e49ec870bcce9517ef12f" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 44235 } } peers { permanent_uuid: "60e8576e4d1d4ceaa4483102024efe5e" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 38259 } } peers { permanent_uuid: "298e8ee11034451391a1c69810f14cce" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 33459 } } }
I20240501 19:59:38.372331 25060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [sys.catalog]: This master's current role is: LEADER
I20240501 19:59:38.372642 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:59:38.372879 25182 catalog_manager.cc:1347] Generated new cluster ID: 57b82506f8514d39a8c9c5812a3f9dd5
I20240501 19:59:38.373200 25182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:59:38.374581 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:59:38.373171 25060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "298e8ee11034451391a1c69810f14cce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933994e2d10e49ec870bcce9517ef12f" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 44235 } } peers { permanent_uuid: "60e8576e4d1d4ceaa4483102024efe5e" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 38259 } } peers { permanent_uuid: "298e8ee11034451391a1c69810f14cce" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 33459 } } }
I20240501 19:59:38.376358 25060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [sys.catalog]: This master's current role is: LEADER
I20240501 19:59:38.378051 25061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60e8576e4d1d4ceaa4483102024efe5e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "298e8ee11034451391a1c69810f14cce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933994e2d10e49ec870bcce9517ef12f" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 44235 } } peers { permanent_uuid: "60e8576e4d1d4ceaa4483102024efe5e" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 38259 } } peers { permanent_uuid: "298e8ee11034451391a1c69810f14cce" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 33459 } } }
I20240501 19:59:38.378752 25061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60e8576e4d1d4ceaa4483102024efe5e [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:59:38.387362 25062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 933994e2d10e49ec870bcce9517ef12f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "298e8ee11034451391a1c69810f14cce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933994e2d10e49ec870bcce9517ef12f" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 44235 } } peers { permanent_uuid: "60e8576e4d1d4ceaa4483102024efe5e" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 38259 } } peers { permanent_uuid: "298e8ee11034451391a1c69810f14cce" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 33459 } } }
I20240501 19:59:38.388348 25062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 933994e2d10e49ec870bcce9517ef12f [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:59:38.392874 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:59:38.400388 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:59:38.400660 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:38.400947 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:59:38.401154 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:38.414201 25182 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:59:38.416795 25182 catalog_manager.cc:1504] Loading token signing keys...
W20240501 19:59:38.449095 25000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.69.190:44235, user_credentials={real_user=slave}} blocked reactor thread for 44610.2us
I20240501 19:59:38.473367 25182 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce: Generated new TSK 0
I20240501 19:59:38.474893 25182 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:59:38.547952 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.130:34889
I20240501 19:59:38.548044 25258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.130:34889 every 8 connection(s)
I20240501 19:59:38.572178 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 19:59:38.577056 25260 heartbeater.cc:347] Connected to a master server at 127.24.69.189:38259
I20240501 19:59:38.577637 25260 heartbeater.cc:464] Registering TS with master...
I20240501 19:59:38.579806 25260 heartbeater.cc:511] Master 127.24.69.189:38259 requested a full tablet report, sending...
I20240501 19:59:38.582567 24951 ts_manager.cc:194] Registered new tserver with Master: 12a4ee47452f4a958b6c5eab7223845e (127.24.69.130:34889)
I20240501 19:59:38.587329 25259 heartbeater.cc:347] Connected to a master server at 127.24.69.188:33459
I20240501 19:59:38.587774 25259 heartbeater.cc:464] Registering TS with master...
I20240501 19:59:38.588973 25259 heartbeater.cc:511] Master 127.24.69.188:33459 requested a full tablet report, sending...
I20240501 19:59:38.591586 25016 ts_manager.cc:194] Registered new tserver with Master: 12a4ee47452f4a958b6c5eab7223845e (127.24.69.130:34889)
I20240501 19:59:38.596226 25016 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49256
W20240501 19:59:38.597738 25265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:38.603963 25261 heartbeater.cc:347] Connected to a master server at 127.24.69.190:44235
I20240501 19:59:38.604437 25261 heartbeater.cc:464] Registering TS with master...
I20240501 19:59:38.605460 25261 heartbeater.cc:511] Master 127.24.69.190:44235 requested a full tablet report, sending...
W20240501 19:59:38.606777 25266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:38.609169 24886 ts_manager.cc:194] Registered new tserver with Master: 12a4ee47452f4a958b6c5eab7223845e (127.24.69.130:34889)
I20240501 19:59:38.614089 24854 server_base.cc:1034] running on GCE node
W20240501 19:59:38.614786 25268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:38.615814 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:59:38.616087 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:59:38.616322 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593578616303 us; error 0 us; skew 500 ppm
I20240501 19:59:38.616994 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 19:59:38.619704 24854 webserver.cc:460] Webserver started at http://127.24.69.131:43629/ using document root <none> and password file <none>
I20240501 19:59:38.620312 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 19:59:38.620549 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:59:38.620868 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:59:38.622212 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-2-root/instance:
uuid: "bb2b68aac97e47f588db4900dc7d6e88"
format_stamp: "Formatted at 2024-05-01 19:59:38 on dist-test-slave-z8x3"
I20240501 19:59:38.627671 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 19:59:38.631695 25273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:38.632531 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:59:38.632867 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-2-root
uuid: "bb2b68aac97e47f588db4900dc7d6e88"
format_stamp: "Formatted at 2024-05-01 19:59:38 on dist-test-slave-z8x3"
I20240501 19:59:38.633203 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:59:38.670627 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:59:38.672715 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:59:38.675115 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:59:38.679311 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:59:38.679574 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:38.679874 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:59:38.680084 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:38.772024 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.131:40821
I20240501 19:59:38.772126 25335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.131:40821 every 8 connection(s)
I20240501 19:59:38.797483 25336 heartbeater.cc:347] Connected to a master server at 127.24.69.188:33459
I20240501 19:59:38.798032 25336 heartbeater.cc:464] Registering TS with master...
I20240501 19:59:38.798041 25337 heartbeater.cc:347] Connected to a master server at 127.24.69.189:38259
I20240501 19:59:38.798719 25337 heartbeater.cc:464] Registering TS with master...
I20240501 19:59:38.799096 25336 heartbeater.cc:511] Master 127.24.69.188:33459 requested a full tablet report, sending...
I20240501 19:59:38.799589 25337 heartbeater.cc:511] Master 127.24.69.189:38259 requested a full tablet report, sending...
I20240501 19:59:38.801249 25016 ts_manager.cc:194] Registered new tserver with Master: bb2b68aac97e47f588db4900dc7d6e88 (127.24.69.131:40821)
I20240501 19:59:38.802742 24951 ts_manager.cc:194] Registered new tserver with Master: bb2b68aac97e47f588db4900dc7d6e88 (127.24.69.131:40821)
I20240501 19:59:38.803390 25016 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49266
I20240501 19:59:38.808322 25338 heartbeater.cc:347] Connected to a master server at 127.24.69.190:44235
I20240501 19:59:38.808621 25338 heartbeater.cc:464] Registering TS with master...
I20240501 19:59:38.809127 25338 heartbeater.cc:511] Master 127.24.69.190:44235 requested a full tablet report, sending...
I20240501 19:59:38.811107 24886 ts_manager.cc:194] Registered new tserver with Master: bb2b68aac97e47f588db4900dc7d6e88 (127.24.69.131:40821)
I20240501 19:59:38.811821 24854 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.020199964s
I20240501 19:59:39.012790 25090 catalog_manager.cc:1259] Loaded cluster ID: 57b82506f8514d39a8c9c5812a3f9dd5
I20240501 19:59:39.013116 25090 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 933994e2d10e49ec870bcce9517ef12f: loading cluster ID for follower catalog manager: success
I20240501 19:59:39.018982 25090 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 933994e2d10e49ec870bcce9517ef12f: acquiring CA information for follower catalog manager: success
I20240501 19:59:39.020141 25096 catalog_manager.cc:1259] Loaded cluster ID: 57b82506f8514d39a8c9c5812a3f9dd5
I20240501 19:59:39.020469 25096 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 60e8576e4d1d4ceaa4483102024efe5e: loading cluster ID for follower catalog manager: success
I20240501 19:59:39.024322 25090 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 933994e2d10e49ec870bcce9517ef12f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 19:59:39.026427 25096 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 60e8576e4d1d4ceaa4483102024efe5e: acquiring CA information for follower catalog manager: success
I20240501 19:59:39.029634 25096 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 60e8576e4d1d4ceaa4483102024efe5e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 19:59:39.281541 25016 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49248
I20240501 19:59:39.601215 25259 heartbeater.cc:503] Master 127.24.69.188:33459 was elected leader, sending a full tablet report...
I20240501 19:59:39.806210 25336 heartbeater.cc:503] Master 127.24.69.188:33459 was elected leader, sending a full tablet report...
I20240501 19:59:39.998859 25092 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:59:40.120975 24854 test_util.cc:268] Using random seed: -314115212
I20240501 19:59:40.178462 25016 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49274:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:59:40.261899 25224 tablet_service.cc:1467] Processing CreateTablet for tablet 8c04e2d690514661a96a22cac985d897 (DEFAULT_TABLE table=test-workload [id=8527ca6df1cf48da9a72145d59307cc8]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240501 19:59:40.263487 25224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c04e2d690514661a96a22cac985d897. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:40.269140 25301 tablet_service.cc:1467] Processing CreateTablet for tablet efe8859c9fb44e438d7debea4a6918ea (DEFAULT_TABLE table=test-workload [id=8527ca6df1cf48da9a72145d59307cc8]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240501 19:59:40.269062 25133 tablet_service.cc:1467] Processing CreateTablet for tablet db107eb02b56486084d4228957462614 (DEFAULT_TABLE table=test-workload [id=8527ca6df1cf48da9a72145d59307cc8]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240501 19:59:40.271883 25133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db107eb02b56486084d4228957462614. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:40.271620 25132 tablet_service.cc:1467] Processing CreateTablet for tablet e127489bcd8b4e19832c390058215f80 (DEFAULT_TABLE table=test-workload [id=8527ca6df1cf48da9a72145d59307cc8]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240501 19:59:40.270776 25301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efe8859c9fb44e438d7debea4a6918ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:40.273092 25132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e127489bcd8b4e19832c390058215f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:40.283690 25168 heartbeater.cc:503] Master 127.24.69.188:33459 was elected leader, sending a full tablet report...
I20240501 19:59:40.289204 25371 tablet_bootstrap.cc:492] T 8c04e2d690514661a96a22cac985d897 P 12a4ee47452f4a958b6c5eab7223845e: Bootstrap starting.
I20240501 19:59:40.291251 25372 tablet_bootstrap.cc:492] T efe8859c9fb44e438d7debea4a6918ea P bb2b68aac97e47f588db4900dc7d6e88: Bootstrap starting.
I20240501 19:59:40.298429 25371 tablet_bootstrap.cc:654] T 8c04e2d690514661a96a22cac985d897 P 12a4ee47452f4a958b6c5eab7223845e: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:40.301170 25373 tablet_bootstrap.cc:492] T db107eb02b56486084d4228957462614 P 73bfb5057eaa4f7f96d6a5f91931add1: Bootstrap starting.
I20240501 19:59:40.305119 25372 tablet_bootstrap.cc:654] T efe8859c9fb44e438d7debea4a6918ea P bb2b68aac97e47f588db4900dc7d6e88: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:40.306512 25371 tablet_bootstrap.cc:492] T 8c04e2d690514661a96a22cac985d897 P 12a4ee47452f4a958b6c5eab7223845e: No bootstrap required, opened a new log
I20240501 19:59:40.306911 25371 ts_tablet_manager.cc:1367] T 8c04e2d690514661a96a22cac985d897 P 12a4ee47452f4a958b6c5eab7223845e: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.007s
I20240501 19:59:40.308676 25373 tablet_bootstrap.cc:654] T db107eb02b56486084d4228957462614 P 73bfb5057eaa4f7f96d6a5f91931add1: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:40.309867 25371 raft_consensus.cc:348] T 8c04e2d690514661a96a22cac985d897 P 12a4ee47452f4a958b6c5eab7223845e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a4ee47452f4a958b6c5eab7223845e" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 34889 } }
I20240501 19:59:40.310525 25371 raft_consensus.cc:374] T 8c04e2d690514661a96a22cac985d897 P 12a4ee47452f4a958b6c5eab7223845e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:40.310834 25371 raft_consensus.cc:729] T 8c04e2d690514661a96a22cac985d897 P 12a4ee47452f4a958b6c5eab7223845e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a4ee47452f4a958b6c5eab7223845e, State: Initialized, Role: FOLLOWER
I20240501 19:59:40.311434 25371 consensus_queue.cc:260] T 8c04e2d690514661a96a22cac985d897 P 12a4ee47452f4a958b6c5eab7223845e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a4ee47452f4a958b6c5eab7223845e" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 34889 } }
I20240501 19:59:40.311962 25371 raft_consensus.cc:388] T 8c04e2d690514661a96a22cac985d897 P 12a4ee47452f4a958b6c5eab7223845e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:59:40.312206 25371 raft_consensus.cc:482] T 8c04e2d690514661a96a22cac985d897 P 12a4ee47452f4a958b6c5eab7223845e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:59:40.312439 25372 tablet_bootstrap.cc:492] T efe8859c9fb44e438d7debea4a6918ea P bb2b68aac97e47f588db4900dc7d6e88: No bootstrap required, opened a new log
I20240501 19:59:40.312780 25371 raft_consensus.cc:3037] T 8c04e2d690514661a96a22cac985d897 P 12a4ee47452f4a958b6c5eab7223845e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:40.313158 25372 ts_tablet_manager.cc:1367] T efe8859c9fb44e438d7debea4a6918ea P bb2b68aac97e47f588db4900dc7d6e88: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.004s
I20240501 19:59:40.315562 25373 tablet_bootstrap.cc:492] T db107eb02b56486084d4228957462614 P 73bfb5057eaa4f7f96d6a5f91931add1: No bootstrap required, opened a new log
I20240501 19:59:40.315619 25372 raft_consensus.cc:348] T efe8859c9fb44e438d7debea4a6918ea P bb2b68aac97e47f588db4900dc7d6e88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2b68aac97e47f588db4900dc7d6e88" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 40821 } }
I20240501 19:59:40.316144 25373 ts_tablet_manager.cc:1367] T db107eb02b56486084d4228957462614 P 73bfb5057eaa4f7f96d6a5f91931add1: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.006s
I20240501 19:59:40.316305 25372 raft_consensus.cc:374] T efe8859c9fb44e438d7debea4a6918ea P bb2b68aac97e47f588db4900dc7d6e88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:40.316612 25372 raft_consensus.cc:729] T efe8859c9fb44e438d7debea4a6918ea P bb2b68aac97e47f588db4900dc7d6e88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb2b68aac97e47f588db4900dc7d6e88, State: Initialized, Role: FOLLOWER
I20240501 19:59:40.317263 25372 consensus_queue.cc:260] T efe8859c9fb44e438d7debea4a6918ea P bb2b68aac97e47f588db4900dc7d6e88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2b68aac97e47f588db4900dc7d6e88" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 40821 } }
I20240501 19:59:40.317898 25372 raft_consensus.cc:388] T efe8859c9fb44e438d7debea4a6918ea P bb2b68aac97e47f588db4900dc7d6e88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:59:40.318193 25372 raft_consensus.cc:482] T efe8859c9fb44e438d7debea4a6918ea P bb2b68aac97e47f588db4900dc7d6e88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:59:40.318527 25372 raft_consensus.cc:3037] T efe8859c9fb44e438d7debea4a6918ea P bb2b68aac97e47f588db4900dc7d6e88 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:40.318673 25371 raft_consensus.cc:504] T 8c04e2d690514661a96a22cac985d897 P 12a4ee47452f4a958b6c5eab7223845e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a4ee47452f4a958b6c5eab7223845e" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 34889 } }
I20240501 19:59:40.318653 25373 raft_consensus.cc:348] T db107eb02b56486084d4228957462614 P 73bfb5057eaa4f7f96d6a5f91931add1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73bfb5057eaa4f7f96d6a5f91931add1" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34261 } }
I20240501 19:59:40.319418 25373 raft_consensus.cc:374] T db107eb02b56486084d4228957462614 P 73bfb5057eaa4f7f96d6a5f91931add1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:40.319463 25371 leader_election.cc:302] T 8c04e2d690514661a96a22cac985d897 P 12a4ee47452f4a958b6c5eab7223845e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12a4ee47452f4a958b6c5eab7223845e; no voters:
I20240501 19:59:40.319829 25373 raft_consensus.cc:729] T db107eb02b56486084d4228957462614 P 73bfb5057eaa4f7f96d6a5f91931add1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73bfb5057eaa4f7f96d6a5f91931add1, State: Initialized, Role: FOLLOWER
I20240501 19:59:40.320995 25373 consensus_queue.cc:260] T db107eb02b56486084d4228957462614 P 73bfb5057eaa4f7f96d6a5f91931add1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73bfb5057eaa4f7f96d6a5f91931add1" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34261 } }
I20240501 19:59:40.321707 25373 raft_consensus.cc:388] T db107eb02b56486084d4228957462614 P 73bfb5057eaa4f7f96d6a5f91931add1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:59:40.322018 25373 raft_consensus.cc:482] T db107eb02b56486084d4228957462614 P 73bfb5057eaa4f7f96d6a5f91931add1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:59:40.322397 25373 raft_consensus.cc:3037] T db107eb02b56486084d4228957462614 P 73bfb5057eaa4f7f96d6a5f91931add1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:40.323228 25371 leader_election.cc:288] T 8c04e2d690514661a96a22cac985d897 P 12a4ee47452f4a958b6c5eab7223845e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:59:40.323737 25377 raft_consensus.cc:2781] T 8c04e2d690514661a96a22cac985d897 P 12a4ee47452f4a958b6c5eab7223845e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:59:40.325412 25372 raft_consensus.cc:504] T efe8859c9fb44e438d7debea4a6918ea P bb2b68aac97e47f588db4900dc7d6e88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2b68aac97e47f588db4900dc7d6e88" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 40821 } }
I20240501 19:59:40.326136 25372 leader_election.cc:302] T efe8859c9fb44e438d7debea4a6918ea P bb2b68aac97e47f588db4900dc7d6e88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb2b68aac97e47f588db4900dc7d6e88; no voters:
I20240501 19:59:40.329665 25373 raft_consensus.cc:504] T db107eb02b56486084d4228957462614 P 73bfb5057eaa4f7f96d6a5f91931add1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73bfb5057eaa4f7f96d6a5f91931add1" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34261 } }
I20240501 19:59:40.330612 25373 leader_election.cc:302] T db107eb02b56486084d4228957462614 P 73bfb5057eaa4f7f96d6a5f91931add1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73bfb5057eaa4f7f96d6a5f91931add1; no voters:
I20240501 19:59:40.334223 25371 ts_tablet_manager.cc:1398] T 8c04e2d690514661a96a22cac985d897 P 12a4ee47452f4a958b6c5eab7223845e: Time spent starting tablet: real 0.027s user 0.025s sys 0.002s
I20240501 19:59:40.336546 25377 raft_consensus.cc:686] T 8c04e2d690514661a96a22cac985d897 P 12a4ee47452f4a958b6c5eab7223845e [term 1 LEADER]: Becoming Leader. State: Replica: 12a4ee47452f4a958b6c5eab7223845e, State: Running, Role: LEADER
I20240501 19:59:40.337155 25377 consensus_queue.cc:237] T 8c04e2d690514661a96a22cac985d897 P 12a4ee47452f4a958b6c5eab7223845e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a4ee47452f4a958b6c5eab7223845e" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 34889 } }
I20240501 19:59:40.338243 25373 leader_election.cc:288] T db107eb02b56486084d4228957462614 P 73bfb5057eaa4f7f96d6a5f91931add1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:59:40.340065 25381 raft_consensus.cc:2781] T db107eb02b56486084d4228957462614 P 73bfb5057eaa4f7f96d6a5f91931add1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:59:40.341992 25372 leader_election.cc:288] T efe8859c9fb44e438d7debea4a6918ea P bb2b68aac97e47f588db4900dc7d6e88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:59:40.343602 25380 raft_consensus.cc:2781] T efe8859c9fb44e438d7debea4a6918ea P bb2b68aac97e47f588db4900dc7d6e88 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:59:40.345427 25373 ts_tablet_manager.cc:1398] T db107eb02b56486084d4228957462614 P 73bfb5057eaa4f7f96d6a5f91931add1: Time spent starting tablet: real 0.029s user 0.014s sys 0.007s
I20240501 19:59:40.346661 25373 tablet_bootstrap.cc:492] T e127489bcd8b4e19832c390058215f80 P 73bfb5057eaa4f7f96d6a5f91931add1: Bootstrap starting.
I20240501 19:59:40.352622 25381 raft_consensus.cc:686] T db107eb02b56486084d4228957462614 P 73bfb5057eaa4f7f96d6a5f91931add1 [term 1 LEADER]: Becoming Leader. State: Replica: 73bfb5057eaa4f7f96d6a5f91931add1, State: Running, Role: LEADER
I20240501 19:59:40.358616 25381 consensus_queue.cc:237] T db107eb02b56486084d4228957462614 P 73bfb5057eaa4f7f96d6a5f91931add1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73bfb5057eaa4f7f96d6a5f91931add1" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34261 } }
I20240501 19:59:40.368578 25373 tablet_bootstrap.cc:654] T e127489bcd8b4e19832c390058215f80 P 73bfb5057eaa4f7f96d6a5f91931add1: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:40.368634 25380 raft_consensus.cc:686] T efe8859c9fb44e438d7debea4a6918ea P bb2b68aac97e47f588db4900dc7d6e88 [term 1 LEADER]: Becoming Leader. State: Replica: bb2b68aac97e47f588db4900dc7d6e88, State: Running, Role: LEADER
I20240501 19:59:40.370187 25380 consensus_queue.cc:237] T efe8859c9fb44e438d7debea4a6918ea P bb2b68aac97e47f588db4900dc7d6e88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2b68aac97e47f588db4900dc7d6e88" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 40821 } }
I20240501 19:59:40.371942 25372 ts_tablet_manager.cc:1398] T efe8859c9fb44e438d7debea4a6918ea P bb2b68aac97e47f588db4900dc7d6e88: Time spent starting tablet: real 0.058s user 0.012s sys 0.028s
I20240501 19:59:40.382237 25015 catalog_manager.cc:5484] T 8c04e2d690514661a96a22cac985d897 P 12a4ee47452f4a958b6c5eab7223845e reported cstate change: term changed from 0 to 1, leader changed from <none> to 12a4ee47452f4a958b6c5eab7223845e (127.24.69.130). New cstate: current_term: 1 leader_uuid: "12a4ee47452f4a958b6c5eab7223845e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a4ee47452f4a958b6c5eab7223845e" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 34889 } health_report { overall_health: HEALTHY } } }
I20240501 19:59:40.388583 25016 catalog_manager.cc:5484] T db107eb02b56486084d4228957462614 P 73bfb5057eaa4f7f96d6a5f91931add1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73bfb5057eaa4f7f96d6a5f91931add1 (127.24.69.129). New cstate: current_term: 1 leader_uuid: "73bfb5057eaa4f7f96d6a5f91931add1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73bfb5057eaa4f7f96d6a5f91931add1" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34261 } health_report { overall_health: HEALTHY } } }
I20240501 19:59:40.399820 25373 tablet_bootstrap.cc:492] T e127489bcd8b4e19832c390058215f80 P 73bfb5057eaa4f7f96d6a5f91931add1: No bootstrap required, opened a new log
I20240501 19:59:40.400362 25373 ts_tablet_manager.cc:1367] T e127489bcd8b4e19832c390058215f80 P 73bfb5057eaa4f7f96d6a5f91931add1: Time spent bootstrapping tablet: real 0.054s user 0.016s sys 0.016s
I20240501 19:59:40.407112 25014 catalog_manager.cc:5484] T efe8859c9fb44e438d7debea4a6918ea P bb2b68aac97e47f588db4900dc7d6e88 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb2b68aac97e47f588db4900dc7d6e88 (127.24.69.131). New cstate: current_term: 1 leader_uuid: "bb2b68aac97e47f588db4900dc7d6e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2b68aac97e47f588db4900dc7d6e88" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 40821 } health_report { overall_health: HEALTHY } } }
I20240501 19:59:40.427564 25373 raft_consensus.cc:348] T e127489bcd8b4e19832c390058215f80 P 73bfb5057eaa4f7f96d6a5f91931add1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73bfb5057eaa4f7f96d6a5f91931add1" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34261 } }
I20240501 19:59:40.430348 25373 raft_consensus.cc:374] T e127489bcd8b4e19832c390058215f80 P 73bfb5057eaa4f7f96d6a5f91931add1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:40.430739 25373 raft_consensus.cc:729] T e127489bcd8b4e19832c390058215f80 P 73bfb5057eaa4f7f96d6a5f91931add1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73bfb5057eaa4f7f96d6a5f91931add1, State: Initialized, Role: FOLLOWER
I20240501 19:59:40.431654 25373 consensus_queue.cc:260] T e127489bcd8b4e19832c390058215f80 P 73bfb5057eaa4f7f96d6a5f91931add1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73bfb5057eaa4f7f96d6a5f91931add1" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34261 } }
I20240501 19:59:40.432406 25373 raft_consensus.cc:388] T e127489bcd8b4e19832c390058215f80 P 73bfb5057eaa4f7f96d6a5f91931add1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:59:40.432757 25373 raft_consensus.cc:482] T e127489bcd8b4e19832c390058215f80 P 73bfb5057eaa4f7f96d6a5f91931add1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:59:40.433153 25373 raft_consensus.cc:3037] T e127489bcd8b4e19832c390058215f80 P 73bfb5057eaa4f7f96d6a5f91931add1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:40.442915 25373 raft_consensus.cc:504] T e127489bcd8b4e19832c390058215f80 P 73bfb5057eaa4f7f96d6a5f91931add1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73bfb5057eaa4f7f96d6a5f91931add1" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34261 } }
I20240501 19:59:40.444200 25373 leader_election.cc:302] T e127489bcd8b4e19832c390058215f80 P 73bfb5057eaa4f7f96d6a5f91931add1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73bfb5057eaa4f7f96d6a5f91931add1; no voters:
I20240501 19:59:40.445066 25381 raft_consensus.cc:2781] T e127489bcd8b4e19832c390058215f80 P 73bfb5057eaa4f7f96d6a5f91931add1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:59:40.445622 25381 raft_consensus.cc:686] T e127489bcd8b4e19832c390058215f80 P 73bfb5057eaa4f7f96d6a5f91931add1 [term 1 LEADER]: Becoming Leader. State: Replica: 73bfb5057eaa4f7f96d6a5f91931add1, State: Running, Role: LEADER
I20240501 19:59:40.446288 25381 consensus_queue.cc:237] T e127489bcd8b4e19832c390058215f80 P 73bfb5057eaa4f7f96d6a5f91931add1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73bfb5057eaa4f7f96d6a5f91931add1" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34261 } }
I20240501 19:59:40.448616 25373 leader_election.cc:288] T e127489bcd8b4e19832c390058215f80 P 73bfb5057eaa4f7f96d6a5f91931add1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:59:40.450757 25373 ts_tablet_manager.cc:1398] T e127489bcd8b4e19832c390058215f80 P 73bfb5057eaa4f7f96d6a5f91931add1: Time spent starting tablet: real 0.050s user 0.013s sys 0.002s
I20240501 19:59:40.458700 25014 catalog_manager.cc:5484] T e127489bcd8b4e19832c390058215f80 P 73bfb5057eaa4f7f96d6a5f91931add1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73bfb5057eaa4f7f96d6a5f91931add1 (127.24.69.129). New cstate: current_term: 1 leader_uuid: "73bfb5057eaa4f7f96d6a5f91931add1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73bfb5057eaa4f7f96d6a5f91931add1" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34261 } health_report { overall_health: HEALTHY } } }
I20240501 19:59:40.513625 24854 tablet_server.cc:178] TabletServer@127.24.69.129:0 shutting down...
I20240501 19:59:40.531466 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:59:40.532279 24854 tablet_replica.cc:330] T e127489bcd8b4e19832c390058215f80 P 73bfb5057eaa4f7f96d6a5f91931add1: stopping tablet replica
I20240501 19:59:40.532891 24854 raft_consensus.cc:2229] T e127489bcd8b4e19832c390058215f80 P 73bfb5057eaa4f7f96d6a5f91931add1 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:59:40.533339 24854 raft_consensus.cc:2258] T e127489bcd8b4e19832c390058215f80 P 73bfb5057eaa4f7f96d6a5f91931add1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:40.535885 24854 tablet_replica.cc:330] T db107eb02b56486084d4228957462614 P 73bfb5057eaa4f7f96d6a5f91931add1: stopping tablet replica
I20240501 19:59:40.536429 24854 raft_consensus.cc:2229] T db107eb02b56486084d4228957462614 P 73bfb5057eaa4f7f96d6a5f91931add1 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:59:40.536798 24854 raft_consensus.cc:2258] T db107eb02b56486084d4228957462614 P 73bfb5057eaa4f7f96d6a5f91931add1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:40.684417 24854 tablet_server.cc:194] TabletServer@127.24.69.129:0 shutdown complete.
I20240501 19:59:40.693128 24854 tablet_server.cc:178] TabletServer@127.24.69.130:0 shutting down...
I20240501 19:59:40.707152 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:59:40.707876 24854 tablet_replica.cc:330] T 8c04e2d690514661a96a22cac985d897 P 12a4ee47452f4a958b6c5eab7223845e: stopping tablet replica
I20240501 19:59:40.708473 24854 raft_consensus.cc:2229] T 8c04e2d690514661a96a22cac985d897 P 12a4ee47452f4a958b6c5eab7223845e [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:59:40.708886 24854 raft_consensus.cc:2258] T 8c04e2d690514661a96a22cac985d897 P 12a4ee47452f4a958b6c5eab7223845e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:40.727286 24854 tablet_server.cc:194] TabletServer@127.24.69.130:0 shutdown complete.
I20240501 19:59:40.734036 24854 tablet_server.cc:178] TabletServer@127.24.69.131:0 shutting down...
I20240501 19:59:40.748908 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:59:40.749608 24854 tablet_replica.cc:330] T efe8859c9fb44e438d7debea4a6918ea P bb2b68aac97e47f588db4900dc7d6e88: stopping tablet replica
I20240501 19:59:40.750118 24854 raft_consensus.cc:2229] T efe8859c9fb44e438d7debea4a6918ea P bb2b68aac97e47f588db4900dc7d6e88 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:59:40.750494 24854 raft_consensus.cc:2258] T efe8859c9fb44e438d7debea4a6918ea P bb2b68aac97e47f588db4900dc7d6e88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:40.769366 24854 tablet_server.cc:194] TabletServer@127.24.69.131:0 shutdown complete.
I20240501 19:59:40.776743 24854 master.cc:537] Master@127.24.69.190:44235 shutting down...
I20240501 19:59:40.875535 24854 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 933994e2d10e49ec870bcce9517ef12f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:59:40.876093 24854 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 933994e2d10e49ec870bcce9517ef12f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:40.876484 24854 tablet_replica.cc:330] T 00000000000000000000000000000000 P 933994e2d10e49ec870bcce9517ef12f: stopping tablet replica
I20240501 19:59:40.894178 24854 master.cc:552] Master@127.24.69.190:44235 shutdown complete.
I20240501 19:59:40.929546 24854 master.cc:537] Master@127.24.69.189:38259 shutting down...
I20240501 19:59:40.941854 24854 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 60e8576e4d1d4ceaa4483102024efe5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:59:40.942481 24854 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 60e8576e4d1d4ceaa4483102024efe5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:40.942881 24854 tablet_replica.cc:330] T 00000000000000000000000000000000 P 60e8576e4d1d4ceaa4483102024efe5e: stopping tablet replica
W20240501 19:59:40.970798 25003 consensus_peers.cc:485] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce -> Peer 60e8576e4d1d4ceaa4483102024efe5e (127.24.69.189:38259): Couldn't send request to peer 60e8576e4d1d4ceaa4483102024efe5e. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:59:40.975060 25000 consensus_peers.cc:485] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce -> Peer 933994e2d10e49ec870bcce9517ef12f (127.24.69.190:44235): Couldn't send request to peer 933994e2d10e49ec870bcce9517ef12f. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.190:44235: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:59:40.984942 24854 master.cc:552] Master@127.24.69.189:38259 shutdown complete.
I20240501 19:59:40.994293 24854 master.cc:537] Master@127.24.69.188:33459 shutting down...
I20240501 19:59:41.007054 24854 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:59:41.007860 24854 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:41.008165 24854 tablet_replica.cc:330] T 00000000000000000000000000000000 P 298e8ee11034451391a1c69810f14cce: stopping tablet replica
I20240501 19:59:41.025566 24854 master.cc:552] Master@127.24.69.188:33459 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (4789 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240501 19:59:41.050727 24854 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.69.190:37003
I20240501 19:59:41.051635 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:59:41.056546 25398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:41.057332 25399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:41.058895 24854 server_base.cc:1034] running on GCE node
W20240501 19:59:41.059463 25401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:41.060154 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:59:41.060382 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:59:41.060539 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593581060520 us; error 0 us; skew 500 ppm
I20240501 19:59:41.061015 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 19:59:41.063218 24854 webserver.cc:460] Webserver started at http://127.24.69.190:42829/ using document root <none> and password file <none>
I20240501 19:59:41.063642 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 19:59:41.063815 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:59:41.064079 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:59:41.065094 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593576104309-24854-0/minicluster-data/master-0-root/instance:
uuid: "b42b983e255d44bdb0554d040121d15f"
format_stamp: "Formatted at 2024-05-01 19:59:41 on dist-test-slave-z8x3"
I20240501 19:59:41.069062 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:59:41.071908 25406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:41.072705 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:59:41.073103 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593576104309-24854-0/minicluster-data/master-0-root
uuid: "b42b983e255d44bdb0554d040121d15f"
format_stamp: "Formatted at 2024-05-01 19:59:41 on dist-test-slave-z8x3"
I20240501 19:59:41.073623 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593576104309-24854-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593576104309-24854-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593576104309-24854-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:59:41.086774 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:59:41.087646 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:59:41.125720 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.190:37003
I20240501 19:59:41.125785 25457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.190:37003 every 8 connection(s)
I20240501 19:59:41.129110 25458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:41.138516 25458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b42b983e255d44bdb0554d040121d15f: Bootstrap starting.
I20240501 19:59:41.141927 25458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b42b983e255d44bdb0554d040121d15f: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:41.145232 25458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b42b983e255d44bdb0554d040121d15f: No bootstrap required, opened a new log
I20240501 19:59:41.146785 25458 raft_consensus.cc:348] T 00000000000000000000000000000000 P b42b983e255d44bdb0554d040121d15f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42b983e255d44bdb0554d040121d15f" member_type: VOTER }
I20240501 19:59:41.147126 25458 raft_consensus.cc:374] T 00000000000000000000000000000000 P b42b983e255d44bdb0554d040121d15f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:41.147296 25458 raft_consensus.cc:729] T 00000000000000000000000000000000 P b42b983e255d44bdb0554d040121d15f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b42b983e255d44bdb0554d040121d15f, State: Initialized, Role: FOLLOWER
I20240501 19:59:41.147720 25458 consensus_queue.cc:260] T 00000000000000000000000000000000 P b42b983e255d44bdb0554d040121d15f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42b983e255d44bdb0554d040121d15f" member_type: VOTER }
I20240501 19:59:41.148116 25458 raft_consensus.cc:388] T 00000000000000000000000000000000 P b42b983e255d44bdb0554d040121d15f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:59:41.148329 25458 raft_consensus.cc:482] T 00000000000000000000000000000000 P b42b983e255d44bdb0554d040121d15f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:59:41.148537 25458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b42b983e255d44bdb0554d040121d15f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:41.151911 25458 raft_consensus.cc:504] T 00000000000000000000000000000000 P b42b983e255d44bdb0554d040121d15f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42b983e255d44bdb0554d040121d15f" member_type: VOTER }
I20240501 19:59:41.152384 25458 leader_election.cc:302] T 00000000000000000000000000000000 P b42b983e255d44bdb0554d040121d15f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b42b983e255d44bdb0554d040121d15f; no voters:
I20240501 19:59:41.153417 25458 leader_election.cc:288] T 00000000000000000000000000000000 P b42b983e255d44bdb0554d040121d15f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:59:41.153687 25461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b42b983e255d44bdb0554d040121d15f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:59:41.154997 25461 raft_consensus.cc:686] T 00000000000000000000000000000000 P b42b983e255d44bdb0554d040121d15f [term 1 LEADER]: Becoming Leader. State: Replica: b42b983e255d44bdb0554d040121d15f, State: Running, Role: LEADER
I20240501 19:59:41.155604 25461 consensus_queue.cc:237] T 00000000000000000000000000000000 P b42b983e255d44bdb0554d040121d15f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42b983e255d44bdb0554d040121d15f" member_type: VOTER }
I20240501 19:59:41.156061 25458 sys_catalog.cc:564] T 00000000000000000000000000000000 P b42b983e255d44bdb0554d040121d15f [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:59:41.158275 25462 sys_catalog.cc:455] T 00000000000000000000000000000000 P b42b983e255d44bdb0554d040121d15f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b42b983e255d44bdb0554d040121d15f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42b983e255d44bdb0554d040121d15f" member_type: VOTER } }
I20240501 19:59:41.158346 25463 sys_catalog.cc:455] T 00000000000000000000000000000000 P b42b983e255d44bdb0554d040121d15f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b42b983e255d44bdb0554d040121d15f. Latest consensus state: current_term: 1 leader_uuid: "b42b983e255d44bdb0554d040121d15f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42b983e255d44bdb0554d040121d15f" member_type: VOTER } }
I20240501 19:59:41.159215 25463 sys_catalog.cc:458] T 00000000000000000000000000000000 P b42b983e255d44bdb0554d040121d15f [sys.catalog]: This master's current role is: LEADER
I20240501 19:59:41.159178 25462 sys_catalog.cc:458] T 00000000000000000000000000000000 P b42b983e255d44bdb0554d040121d15f [sys.catalog]: This master's current role is: LEADER
I20240501 19:59:41.161957 25466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:59:41.169473 25466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:59:41.170341 24854 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 19:59:41.177024 25466 catalog_manager.cc:1347] Generated new cluster ID: fc94271047d04c239e8ef3a4ff829b09
I20240501 19:59:41.177286 25466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:59:41.195959 25466 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:59:41.197130 25466 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:59:41.212709 25466 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b42b983e255d44bdb0554d040121d15f: Generated new TSK 0
I20240501 19:59:41.213277 25466 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:59:41.236570 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:59:41.242591 25479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:41.243829 25480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:41.244711 25482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:41.248035 24854 server_base.cc:1034] running on GCE node
I20240501 19:59:41.249718 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:59:41.249924 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:59:41.250110 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593581250092 us; error 0 us; skew 500 ppm
I20240501 19:59:41.250615 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 19:59:41.252915 24854 webserver.cc:460] Webserver started at http://127.24.69.129:43983/ using document root <none> and password file <none>
I20240501 19:59:41.253353 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 19:59:41.253525 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:59:41.253768 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:59:41.254802 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593576104309-24854-0/minicluster-data/ts-0-root/instance:
uuid: "7f6cabdca8b242d397b00a90d55ef9ec"
format_stamp: "Formatted at 2024-05-01 19:59:41 on dist-test-slave-z8x3"
I20240501 19:59:41.258929 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 19:59:41.261946 25487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:41.262676 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:59:41.263088 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593576104309-24854-0/minicluster-data/ts-0-root
uuid: "7f6cabdca8b242d397b00a90d55ef9ec"
format_stamp: "Formatted at 2024-05-01 19:59:41 on dist-test-slave-z8x3"
I20240501 19:59:41.263428 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593576104309-24854-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593576104309-24854-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593576104309-24854-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:59:41.280468 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:59:41.281344 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:59:41.282668 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:59:41.284793 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:59:41.284979 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:41.285194 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:59:41.285346 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:41.321308 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.129:39987
I20240501 19:59:41.321413 25549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.129:39987 every 8 connection(s)
I20240501 19:59:41.325474 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:59:41.333041 25554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:41.333694 25555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:41.336181 25557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:41.336760 24854 server_base.cc:1034] running on GCE node
I20240501 19:59:41.337657 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:59:41.337863 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:59:41.338024 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593581338009 us; error 0 us; skew 500 ppm
I20240501 19:59:41.338467 25550 heartbeater.cc:347] Connected to a master server at 127.24.69.190:37003
I20240501 19:59:41.338510 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 19:59:41.338913 25550 heartbeater.cc:464] Registering TS with master...
I20240501 19:59:41.339546 25550 heartbeater.cc:511] Master 127.24.69.190:37003 requested a full tablet report, sending...
I20240501 19:59:41.341418 24854 webserver.cc:460] Webserver started at http://127.24.69.130:39363/ using document root <none> and password file <none>
I20240501 19:59:41.341806 25423 ts_manager.cc:194] Registered new tserver with Master: 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129:39987)
I20240501 19:59:41.342022 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 19:59:41.342296 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:59:41.342614 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:59:41.343953 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593576104309-24854-0/minicluster-data/ts-1-root/instance:
uuid: "5d2ab23455b64a75a08d71d0db37be10"
format_stamp: "Formatted at 2024-05-01 19:59:41 on dist-test-slave-z8x3"
I20240501 19:59:41.344112 25423 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55676
I20240501 19:59:41.348129 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240501 19:59:41.351155 25562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:41.351847 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:59:41.352092 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593576104309-24854-0/minicluster-data/ts-1-root
uuid: "5d2ab23455b64a75a08d71d0db37be10"
format_stamp: "Formatted at 2024-05-01 19:59:41 on dist-test-slave-z8x3"
I20240501 19:59:41.352378 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593576104309-24854-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593576104309-24854-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593576104309-24854-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:59:41.381026 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:59:41.382011 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:59:41.383324 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:59:41.385433 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:59:41.385632 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:41.385856 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:59:41.386004 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:41.421375 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.130:33157
I20240501 19:59:41.421461 25624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.130:33157 every 8 connection(s)
I20240501 19:59:41.425936 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:59:41.432825 25628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:41.433862 25629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:41.436826 25631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:41.437296 24854 server_base.cc:1034] running on GCE node
I20240501 19:59:41.437553 25625 heartbeater.cc:347] Connected to a master server at 127.24.69.190:37003
I20240501 19:59:41.437947 25625 heartbeater.cc:464] Registering TS with master...
I20240501 19:59:41.438151 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:59:41.438390 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:59:41.438547 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593581438530 us; error 0 us; skew 500 ppm
I20240501 19:59:41.438617 25625 heartbeater.cc:511] Master 127.24.69.190:37003 requested a full tablet report, sending...
I20240501 19:59:41.439072 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 19:59:41.440593 25423 ts_manager.cc:194] Registered new tserver with Master: 5d2ab23455b64a75a08d71d0db37be10 (127.24.69.130:33157)
I20240501 19:59:41.441613 24854 webserver.cc:460] Webserver started at http://127.24.69.131:37147/ using document root <none> and password file <none>
I20240501 19:59:41.441977 25423 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55692
I20240501 19:59:41.442080 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 19:59:41.442298 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:59:41.442546 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:59:41.443779 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593576104309-24854-0/minicluster-data/ts-2-root/instance:
uuid: "016da4070e314b2786e8614d8a954ecd"
format_stamp: "Formatted at 2024-05-01 19:59:41 on dist-test-slave-z8x3"
I20240501 19:59:41.447857 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 19:59:41.450839 25636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:41.451537 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:59:41.451795 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593576104309-24854-0/minicluster-data/ts-2-root
uuid: "016da4070e314b2786e8614d8a954ecd"
format_stamp: "Formatted at 2024-05-01 19:59:41 on dist-test-slave-z8x3"
I20240501 19:59:41.452046 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593576104309-24854-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593576104309-24854-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593576104309-24854-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:59:41.465066 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:59:41.466012 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:59:41.467271 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:59:41.469381 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:59:41.469558 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:41.469797 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:59:41.469935 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:41.505959 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.131:42353
I20240501 19:59:41.506053 25698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.131:42353 every 8 connection(s)
I20240501 19:59:41.518479 25699 heartbeater.cc:347] Connected to a master server at 127.24.69.190:37003
I20240501 19:59:41.518834 25699 heartbeater.cc:464] Registering TS with master...
I20240501 19:59:41.519505 25699 heartbeater.cc:511] Master 127.24.69.190:37003 requested a full tablet report, sending...
I20240501 19:59:41.521214 25423 ts_manager.cc:194] Registered new tserver with Master: 016da4070e314b2786e8614d8a954ecd (127.24.69.131:42353)
I20240501 19:59:41.522056 24854 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.0132386s
I20240501 19:59:41.522460 25423 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55702
I20240501 19:59:42.346403 25550 heartbeater.cc:503] Master 127.24.69.190:37003 was elected leader, sending a full tablet report...
I20240501 19:59:42.444500 25625 heartbeater.cc:503] Master 127.24.69.190:37003 was elected leader, sending a full tablet report...
I20240501 19:59:42.524798 25699 heartbeater.cc:503] Master 127.24.69.190:37003 was elected leader, sending a full tablet report...
I20240501 19:59:42.553889 24854 test_util.cc:268] Using random seed: -311682292
I20240501 19:59:42.576829 25423 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55716:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 19:59:42.579005 25423 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:59:42.645838 25590 tablet_service.cc:1467] Processing CreateTablet for tablet 1be0bc54d8f942aa8eb1a1cdfc485a7d (DEFAULT_TABLE table=test-workload [id=c12aeeb1d75d4b98b4ba1e0fa9a97eb2]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240501 19:59:42.646555 25587 tablet_service.cc:1467] Processing CreateTablet for tablet 47f6e1c34803471289c79387682db511 (DEFAULT_TABLE table=test-workload [id=c12aeeb1d75d4b98b4ba1e0fa9a97eb2]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240501 19:59:42.646862 25588 tablet_service.cc:1467] Processing CreateTablet for tablet ba5ada435b794cc194e3cbff0d994915 (DEFAULT_TABLE table=test-workload [id=c12aeeb1d75d4b98b4ba1e0fa9a97eb2]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240501 19:59:42.647277 25590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be0bc54d8f942aa8eb1a1cdfc485a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:42.647935 25587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47f6e1c34803471289c79387682db511. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:42.645843 25589 tablet_service.cc:1467] Processing CreateTablet for tablet 3a18557702154a5a91ff0d36e4df951d (DEFAULT_TABLE table=test-workload [id=c12aeeb1d75d4b98b4ba1e0fa9a97eb2]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240501 19:59:42.649408 25589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a18557702154a5a91ff0d36e4df951d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:42.654119 25664 tablet_service.cc:1467] Processing CreateTablet for tablet 1be0bc54d8f942aa8eb1a1cdfc485a7d (DEFAULT_TABLE table=test-workload [id=c12aeeb1d75d4b98b4ba1e0fa9a97eb2]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240501 19:59:42.654958 25588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba5ada435b794cc194e3cbff0d994915. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:42.655404 25664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be0bc54d8f942aa8eb1a1cdfc485a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:42.657491 25663 tablet_service.cc:1467] Processing CreateTablet for tablet 3a18557702154a5a91ff0d36e4df951d (DEFAULT_TABLE table=test-workload [id=c12aeeb1d75d4b98b4ba1e0fa9a97eb2]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240501 19:59:42.657655 25515 tablet_service.cc:1467] Processing CreateTablet for tablet 1be0bc54d8f942aa8eb1a1cdfc485a7d (DEFAULT_TABLE table=test-workload [id=c12aeeb1d75d4b98b4ba1e0fa9a97eb2]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240501 19:59:42.658941 25515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be0bc54d8f942aa8eb1a1cdfc485a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:42.661450 25661 tablet_service.cc:1467] Processing CreateTablet for tablet 47f6e1c34803471289c79387682db511 (DEFAULT_TABLE table=test-workload [id=c12aeeb1d75d4b98b4ba1e0fa9a97eb2]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240501 19:59:42.660049 25662 tablet_service.cc:1467] Processing CreateTablet for tablet ba5ada435b794cc194e3cbff0d994915 (DEFAULT_TABLE table=test-workload [id=c12aeeb1d75d4b98b4ba1e0fa9a97eb2]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240501 19:59:42.665139 25663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a18557702154a5a91ff0d36e4df951d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:42.665789 25661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47f6e1c34803471289c79387682db511. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:42.667572 25660 tablet_service.cc:1467] Processing CreateTablet for tablet 38c3d8dabd6648ff9a4262c784b88c80 (DEFAULT_TABLE table=test-workload [id=c12aeeb1d75d4b98b4ba1e0fa9a97eb2]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240501 19:59:42.667865 25513 tablet_service.cc:1467] Processing CreateTablet for tablet ba5ada435b794cc194e3cbff0d994915 (DEFAULT_TABLE table=test-workload [id=c12aeeb1d75d4b98b4ba1e0fa9a97eb2]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240501 19:59:42.668951 25660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c3d8dabd6648ff9a4262c784b88c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:42.669239 25513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba5ada435b794cc194e3cbff0d994915. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:42.674280 25659 tablet_service.cc:1467] Processing CreateTablet for tablet a528234b11234a6a8516d9365b49d8db (DEFAULT_TABLE table=test-workload [id=c12aeeb1d75d4b98b4ba1e0fa9a97eb2]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240501 19:59:42.675585 25659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a528234b11234a6a8516d9365b49d8db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:42.677028 25585 tablet_service.cc:1467] Processing CreateTablet for tablet 7320413158f34e7d80d98dd88b562af7 (DEFAULT_TABLE table=test-workload [id=c12aeeb1d75d4b98b4ba1e0fa9a97eb2]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240501 19:59:42.677402 25658 tablet_service.cc:1467] Processing CreateTablet for tablet 7320413158f34e7d80d98dd88b562af7 (DEFAULT_TABLE table=test-workload [id=c12aeeb1d75d4b98b4ba1e0fa9a97eb2]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240501 19:59:42.678297 25585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7320413158f34e7d80d98dd88b562af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:42.675868 25584 tablet_service.cc:1467] Processing CreateTablet for tablet e659b732ef82488eac87492a463bd702 (DEFAULT_TABLE table=test-workload [id=c12aeeb1d75d4b98b4ba1e0fa9a97eb2]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240501 19:59:42.678709 25658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7320413158f34e7d80d98dd88b562af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:42.679386 25584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e659b732ef82488eac87492a463bd702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:42.680842 25512 tablet_service.cc:1467] Processing CreateTablet for tablet 47f6e1c34803471289c79387682db511 (DEFAULT_TABLE table=test-workload [id=c12aeeb1d75d4b98b4ba1e0fa9a97eb2]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240501 19:59:42.675866 25586 tablet_service.cc:1467] Processing CreateTablet for tablet a528234b11234a6a8516d9365b49d8db (DEFAULT_TABLE table=test-workload [id=c12aeeb1d75d4b98b4ba1e0fa9a97eb2]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240501 19:59:42.681425 25657 tablet_service.cc:1467] Processing CreateTablet for tablet e659b732ef82488eac87492a463bd702 (DEFAULT_TABLE table=test-workload [id=c12aeeb1d75d4b98b4ba1e0fa9a97eb2]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240501 19:59:42.682184 25512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47f6e1c34803471289c79387682db511. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:42.682269 25586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a528234b11234a6a8516d9365b49d8db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:42.682687 25657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e659b732ef82488eac87492a463bd702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:42.665421 25514 tablet_service.cc:1467] Processing CreateTablet for tablet 3a18557702154a5a91ff0d36e4df951d (DEFAULT_TABLE table=test-workload [id=c12aeeb1d75d4b98b4ba1e0fa9a97eb2]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240501 19:59:42.685063 25514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a18557702154a5a91ff0d36e4df951d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:42.685525 25508 tablet_service.cc:1467] Processing CreateTablet for tablet e659b732ef82488eac87492a463bd702 (DEFAULT_TABLE table=test-workload [id=c12aeeb1d75d4b98b4ba1e0fa9a97eb2]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240501 19:59:42.687271 25508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e659b732ef82488eac87492a463bd702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:42.689623 25662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba5ada435b794cc194e3cbff0d994915. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:42.691227 25510 tablet_service.cc:1467] Processing CreateTablet for tablet a528234b11234a6a8516d9365b49d8db (DEFAULT_TABLE table=test-workload [id=c12aeeb1d75d4b98b4ba1e0fa9a97eb2]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240501 19:59:42.691356 25509 tablet_service.cc:1467] Processing CreateTablet for tablet 7320413158f34e7d80d98dd88b562af7 (DEFAULT_TABLE table=test-workload [id=c12aeeb1d75d4b98b4ba1e0fa9a97eb2]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240501 19:59:42.685458 25511 tablet_service.cc:1467] Processing CreateTablet for tablet 38c3d8dabd6648ff9a4262c784b88c80 (DEFAULT_TABLE table=test-workload [id=c12aeeb1d75d4b98b4ba1e0fa9a97eb2]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240501 19:59:42.693156 25510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a528234b11234a6a8516d9365b49d8db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:42.696777 25511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c3d8dabd6648ff9a4262c784b88c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:42.698045 25509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7320413158f34e7d80d98dd88b562af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:42.699672 25587 tablet_service.cc:1467] Processing CreateTablet for tablet 38c3d8dabd6648ff9a4262c784b88c80 (DEFAULT_TABLE table=test-workload [id=c12aeeb1d75d4b98b4ba1e0fa9a97eb2]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240501 19:59:42.700986 25587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c3d8dabd6648ff9a4262c784b88c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:42.732434 25720 tablet_bootstrap.cc:492] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10: Bootstrap starting.
I20240501 19:59:42.732831 25719 tablet_bootstrap.cc:492] T 47f6e1c34803471289c79387682db511 P 016da4070e314b2786e8614d8a954ecd: Bootstrap starting.
I20240501 19:59:42.736761 25720 tablet_bootstrap.cc:654] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:42.757836 25719 tablet_bootstrap.cc:654] T 47f6e1c34803471289c79387682db511 P 016da4070e314b2786e8614d8a954ecd: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:42.799166 25721 tablet_bootstrap.cc:492] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 7f6cabdca8b242d397b00a90d55ef9ec: Bootstrap starting.
I20240501 19:59:42.803503 25720 tablet_bootstrap.cc:492] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10: No bootstrap required, opened a new log
I20240501 19:59:42.804057 25720 ts_tablet_manager.cc:1367] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10: Time spent bootstrapping tablet: real 0.072s user 0.008s sys 0.013s
I20240501 19:59:42.804363 25719 tablet_bootstrap.cc:492] T 47f6e1c34803471289c79387682db511 P 016da4070e314b2786e8614d8a954ecd: No bootstrap required, opened a new log
I20240501 19:59:42.806056 25719 ts_tablet_manager.cc:1367] T 47f6e1c34803471289c79387682db511 P 016da4070e314b2786e8614d8a954ecd: Time spent bootstrapping tablet: real 0.073s user 0.016s sys 0.006s
I20240501 19:59:42.807260 25720 raft_consensus.cc:348] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } }
I20240501 19:59:42.808120 25720 raft_consensus.cc:374] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:42.808488 25720 raft_consensus.cc:729] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d2ab23455b64a75a08d71d0db37be10, State: Initialized, Role: FOLLOWER
I20240501 19:59:42.808552 25719 raft_consensus.cc:348] T 47f6e1c34803471289c79387682db511 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:42.809366 25719 raft_consensus.cc:374] T 47f6e1c34803471289c79387682db511 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:42.810506 25719 raft_consensus.cc:729] T 47f6e1c34803471289c79387682db511 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 016da4070e314b2786e8614d8a954ecd, State: Initialized, Role: FOLLOWER
I20240501 19:59:42.810591 25720 consensus_queue.cc:260] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } }
I20240501 19:59:42.811201 25719 consensus_queue.cc:260] T 47f6e1c34803471289c79387682db511 P 016da4070e314b2786e8614d8a954ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:42.814532 25719 ts_tablet_manager.cc:1398] T 47f6e1c34803471289c79387682db511 P 016da4070e314b2786e8614d8a954ecd: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20240501 19:59:42.815382 25719 tablet_bootstrap.cc:492] T a528234b11234a6a8516d9365b49d8db P 016da4070e314b2786e8614d8a954ecd: Bootstrap starting.
I20240501 19:59:42.816771 25721 tablet_bootstrap.cc:654] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 7f6cabdca8b242d397b00a90d55ef9ec: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:42.821446 25719 tablet_bootstrap.cc:654] T a528234b11234a6a8516d9365b49d8db P 016da4070e314b2786e8614d8a954ecd: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:42.825605 25720 ts_tablet_manager.cc:1398] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10: Time spent starting tablet: real 0.021s user 0.010s sys 0.000s
I20240501 19:59:42.825752 25721 tablet_bootstrap.cc:492] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 7f6cabdca8b242d397b00a90d55ef9ec: No bootstrap required, opened a new log
I20240501 19:59:42.826445 25721 ts_tablet_manager.cc:1367] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 7f6cabdca8b242d397b00a90d55ef9ec: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.006s
I20240501 19:59:42.826512 25720 tablet_bootstrap.cc:492] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10: Bootstrap starting.
I20240501 19:59:42.828644 25719 tablet_bootstrap.cc:492] T a528234b11234a6a8516d9365b49d8db P 016da4070e314b2786e8614d8a954ecd: No bootstrap required, opened a new log
I20240501 19:59:42.829080 25719 ts_tablet_manager.cc:1367] T a528234b11234a6a8516d9365b49d8db P 016da4070e314b2786e8614d8a954ecd: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.001s
I20240501 19:59:42.829139 25721 raft_consensus.cc:348] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } }
I20240501 19:59:42.830140 25721 raft_consensus.cc:374] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:42.830472 25721 raft_consensus.cc:729] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6cabdca8b242d397b00a90d55ef9ec, State: Initialized, Role: FOLLOWER
I20240501 19:59:42.831288 25721 consensus_queue.cc:260] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 7f6cabdca8b242d397b00a90d55ef9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } }
I20240501 19:59:42.831655 25719 raft_consensus.cc:348] T a528234b11234a6a8516d9365b49d8db P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:42.832238 25720 tablet_bootstrap.cc:654] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:42.832430 25719 raft_consensus.cc:374] T a528234b11234a6a8516d9365b49d8db P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:42.832770 25719 raft_consensus.cc:729] T a528234b11234a6a8516d9365b49d8db P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 016da4070e314b2786e8614d8a954ecd, State: Initialized, Role: FOLLOWER
I20240501 19:59:42.833449 25719 consensus_queue.cc:260] T a528234b11234a6a8516d9365b49d8db P 016da4070e314b2786e8614d8a954ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:42.838270 25719 ts_tablet_manager.cc:1398] T a528234b11234a6a8516d9365b49d8db P 016da4070e314b2786e8614d8a954ecd: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240501 19:59:42.839463 25719 tablet_bootstrap.cc:492] T 3a18557702154a5a91ff0d36e4df951d P 016da4070e314b2786e8614d8a954ecd: Bootstrap starting.
I20240501 19:59:42.846143 25719 tablet_bootstrap.cc:654] T 3a18557702154a5a91ff0d36e4df951d P 016da4070e314b2786e8614d8a954ecd: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:42.849567 25726 raft_consensus.cc:482] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:59:42.850188 25726 raft_consensus.cc:504] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } }
I20240501 19:59:42.863494 25726 leader_election.cc:288] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 016da4070e314b2786e8614d8a954ecd (127.24.69.131:42353), 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129:39987)
I20240501 19:59:42.879271 25721 ts_tablet_manager.cc:1398] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 7f6cabdca8b242d397b00a90d55ef9ec: Time spent starting tablet: real 0.052s user 0.023s sys 0.023s
I20240501 19:59:42.880175 25674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be0bc54d8f942aa8eb1a1cdfc485a7d" candidate_uuid: "5d2ab23455b64a75a08d71d0db37be10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016da4070e314b2786e8614d8a954ecd" is_pre_election: true
I20240501 19:59:42.880796 25719 tablet_bootstrap.cc:492] T 3a18557702154a5a91ff0d36e4df951d P 016da4070e314b2786e8614d8a954ecd: No bootstrap required, opened a new log
I20240501 19:59:42.881283 25719 ts_tablet_manager.cc:1367] T 3a18557702154a5a91ff0d36e4df951d P 016da4070e314b2786e8614d8a954ecd: Time spent bootstrapping tablet: real 0.042s user 0.015s sys 0.023s
W20240501 19:59:42.882102 25564 leader_election.cc:341] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 016da4070e314b2786e8614d8a954ecd (127.24.69.131:42353): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:59:42.883028 25525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be0bc54d8f942aa8eb1a1cdfc485a7d" candidate_uuid: "5d2ab23455b64a75a08d71d0db37be10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" is_pre_election: true
I20240501 19:59:42.883702 25525 raft_consensus.cc:2454] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d2ab23455b64a75a08d71d0db37be10 in term 0.
I20240501 19:59:42.884153 25720 tablet_bootstrap.cc:492] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10: No bootstrap required, opened a new log
I20240501 19:59:42.884244 25721 tablet_bootstrap.cc:492] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec: Bootstrap starting.
I20240501 19:59:42.884809 25720 ts_tablet_manager.cc:1367] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10: Time spent bootstrapping tablet: real 0.058s user 0.024s sys 0.017s
I20240501 19:59:42.884208 25719 raft_consensus.cc:348] T 3a18557702154a5a91ff0d36e4df951d P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:42.885157 25719 raft_consensus.cc:374] T 3a18557702154a5a91ff0d36e4df951d P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:42.885454 25719 raft_consensus.cc:729] T 3a18557702154a5a91ff0d36e4df951d P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 016da4070e314b2786e8614d8a954ecd, State: Initialized, Role: FOLLOWER
I20240501 19:59:42.886144 25719 consensus_queue.cc:260] T 3a18557702154a5a91ff0d36e4df951d P 016da4070e314b2786e8614d8a954ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:42.887269 25720 raft_consensus.cc:348] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:42.887967 25720 raft_consensus.cc:374] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:42.888278 25720 raft_consensus.cc:729] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d2ab23455b64a75a08d71d0db37be10, State: Initialized, Role: FOLLOWER
I20240501 19:59:42.889039 25720 consensus_queue.cc:260] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:42.890015 25721 tablet_bootstrap.cc:654] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:42.890713 25719 ts_tablet_manager.cc:1398] T 3a18557702154a5a91ff0d36e4df951d P 016da4070e314b2786e8614d8a954ecd: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240501 19:59:42.890899 25563 leader_election.cc:302] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d2ab23455b64a75a08d71d0db37be10, 7f6cabdca8b242d397b00a90d55ef9ec; no voters: 016da4070e314b2786e8614d8a954ecd
I20240501 19:59:42.891659 25719 tablet_bootstrap.cc:492] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd: Bootstrap starting.
I20240501 19:59:42.892143 25726 raft_consensus.cc:2781] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:59:42.892554 25726 raft_consensus.cc:482] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:59:42.892979 25726 raft_consensus.cc:3037] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:42.897090 25719 tablet_bootstrap.cc:654] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:42.897212 25720 ts_tablet_manager.cc:1398] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20240501 19:59:42.903002 25720 tablet_bootstrap.cc:492] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10: Bootstrap starting.
I20240501 19:59:42.909485 25720 tablet_bootstrap.cc:654] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:42.913415 25734 raft_consensus.cc:482] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:59:42.913921 25734 raft_consensus.cc:504] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:42.915532 25719 tablet_bootstrap.cc:492] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd: No bootstrap required, opened a new log
I20240501 19:59:42.916002 25719 ts_tablet_manager.cc:1367] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.004s
I20240501 19:59:42.916752 25525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f6e1c34803471289c79387682db511" candidate_uuid: "5d2ab23455b64a75a08d71d0db37be10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" is_pre_election: true
W20240501 19:59:42.918099 25563 leader_election.cc:341] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129:39987): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:59:42.918891 25719 raft_consensus.cc:348] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:42.919584 25719 raft_consensus.cc:374] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:42.919867 25719 raft_consensus.cc:729] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 016da4070e314b2786e8614d8a954ecd, State: Initialized, Role: FOLLOWER
I20240501 19:59:42.920452 25674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f6e1c34803471289c79387682db511" candidate_uuid: "5d2ab23455b64a75a08d71d0db37be10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016da4070e314b2786e8614d8a954ecd" is_pre_election: true
I20240501 19:59:42.920539 25719 consensus_queue.cc:260] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:42.921082 25674 raft_consensus.cc:2454] T 47f6e1c34803471289c79387682db511 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d2ab23455b64a75a08d71d0db37be10 in term 0.
I20240501 19:59:42.922159 25564 leader_election.cc:302] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 016da4070e314b2786e8614d8a954ecd, 5d2ab23455b64a75a08d71d0db37be10; no voters: 7f6cabdca8b242d397b00a90d55ef9ec
I20240501 19:59:42.925509 25719 ts_tablet_manager.cc:1398] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd: Time spent starting tablet: real 0.009s user 0.005s sys 0.002s
I20240501 19:59:42.926784 25719 tablet_bootstrap.cc:492] T e659b732ef82488eac87492a463bd702 P 016da4070e314b2786e8614d8a954ecd: Bootstrap starting.
I20240501 19:59:42.927677 25734 leader_election.cc:288] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129:39987), 016da4070e314b2786e8614d8a954ecd (127.24.69.131:42353)
I20240501 19:59:42.928171 25734 raft_consensus.cc:2781] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:59:42.928526 25734 raft_consensus.cc:482] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:59:42.928846 25734 raft_consensus.cc:3037] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:42.932673 25719 tablet_bootstrap.cc:654] T e659b732ef82488eac87492a463bd702 P 016da4070e314b2786e8614d8a954ecd: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:42.940333 25726 raft_consensus.cc:504] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } }
I20240501 19:59:42.943747 25726 leader_election.cc:288] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 election: Requested vote from peers 016da4070e314b2786e8614d8a954ecd (127.24.69.131:42353), 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129:39987)
I20240501 19:59:42.944414 25525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be0bc54d8f942aa8eb1a1cdfc485a7d" candidate_uuid: "5d2ab23455b64a75a08d71d0db37be10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6cabdca8b242d397b00a90d55ef9ec"
I20240501 19:59:42.945274 25525 raft_consensus.cc:3037] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:42.946537 25721 tablet_bootstrap.cc:492] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec: No bootstrap required, opened a new log
I20240501 19:59:42.946907 25721 ts_tablet_manager.cc:1367] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec: Time spent bootstrapping tablet: real 0.063s user 0.022s sys 0.038s
I20240501 19:59:42.948657 25721 raft_consensus.cc:348] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:42.949149 25721 raft_consensus.cc:374] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:42.949376 25721 raft_consensus.cc:729] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6cabdca8b242d397b00a90d55ef9ec, State: Initialized, Role: FOLLOWER
I20240501 19:59:42.949910 25721 consensus_queue.cc:260] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:42.952245 25721 ts_tablet_manager.cc:1398] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 19:59:42.953083 25721 tablet_bootstrap.cc:492] T 3a18557702154a5a91ff0d36e4df951d P 7f6cabdca8b242d397b00a90d55ef9ec: Bootstrap starting.
I20240501 19:59:42.954708 25674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be0bc54d8f942aa8eb1a1cdfc485a7d" candidate_uuid: "5d2ab23455b64a75a08d71d0db37be10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016da4070e314b2786e8614d8a954ecd"
I20240501 19:59:42.957690 25525 raft_consensus.cc:2454] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d2ab23455b64a75a08d71d0db37be10 in term 1.
I20240501 19:59:42.958631 25721 tablet_bootstrap.cc:654] T 3a18557702154a5a91ff0d36e4df951d P 7f6cabdca8b242d397b00a90d55ef9ec: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:42.959115 25563 leader_election.cc:302] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d2ab23455b64a75a08d71d0db37be10, 7f6cabdca8b242d397b00a90d55ef9ec; no voters:
W20240501 19:59:42.960397 25564 leader_election.cc:341] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 016da4070e314b2786e8614d8a954ecd (127.24.69.131:42353): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:59:42.964347 25726 raft_consensus.cc:2781] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:59:42.965468 25726 raft_consensus.cc:686] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [term 1 LEADER]: Becoming Leader. State: Replica: 5d2ab23455b64a75a08d71d0db37be10, State: Running, Role: LEADER
I20240501 19:59:42.968161 25726 consensus_queue.cc:237] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } }
I20240501 19:59:42.973800 25734 raft_consensus.cc:504] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:42.966514 25720 tablet_bootstrap.cc:492] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10: No bootstrap required, opened a new log
I20240501 19:59:42.975455 25719 tablet_bootstrap.cc:492] T e659b732ef82488eac87492a463bd702 P 016da4070e314b2786e8614d8a954ecd: No bootstrap required, opened a new log
I20240501 19:59:42.975946 25719 ts_tablet_manager.cc:1367] T e659b732ef82488eac87492a463bd702 P 016da4070e314b2786e8614d8a954ecd: Time spent bootstrapping tablet: real 0.049s user 0.030s sys 0.015s
I20240501 19:59:42.976121 25720 ts_tablet_manager.cc:1367] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10: Time spent bootstrapping tablet: real 0.073s user 0.014s sys 0.003s
I20240501 19:59:42.976406 25525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f6e1c34803471289c79387682db511" candidate_uuid: "5d2ab23455b64a75a08d71d0db37be10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6cabdca8b242d397b00a90d55ef9ec"
W20240501 19:59:42.977699 25563 leader_election.cc:341] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129:39987): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:59:42.978403 25719 raft_consensus.cc:348] T e659b732ef82488eac87492a463bd702 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:42.979120 25719 raft_consensus.cc:374] T e659b732ef82488eac87492a463bd702 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:42.979415 25719 raft_consensus.cc:729] T e659b732ef82488eac87492a463bd702 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 016da4070e314b2786e8614d8a954ecd, State: Initialized, Role: FOLLOWER
I20240501 19:59:42.979516 25674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f6e1c34803471289c79387682db511" candidate_uuid: "5d2ab23455b64a75a08d71d0db37be10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016da4070e314b2786e8614d8a954ecd"
I20240501 19:59:42.980103 25674 raft_consensus.cc:3037] T 47f6e1c34803471289c79387682db511 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:42.980240 25719 consensus_queue.cc:260] T e659b732ef82488eac87492a463bd702 P 016da4070e314b2786e8614d8a954ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:42.980300 25720 raft_consensus.cc:348] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:42.982666 25720 raft_consensus.cc:374] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:42.983574 25720 raft_consensus.cc:729] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d2ab23455b64a75a08d71d0db37be10, State: Initialized, Role: FOLLOWER
I20240501 19:59:42.985051 25720 consensus_queue.cc:260] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:42.987277 25674 raft_consensus.cc:2454] T 47f6e1c34803471289c79387682db511 P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d2ab23455b64a75a08d71d0db37be10 in term 1.
I20240501 19:59:42.987659 25734 leader_election.cc:288] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 election: Requested vote from peers 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129:39987), 016da4070e314b2786e8614d8a954ecd (127.24.69.131:42353)
I20240501 19:59:42.988914 25564 leader_election.cc:302] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 016da4070e314b2786e8614d8a954ecd, 5d2ab23455b64a75a08d71d0db37be10; no voters: 7f6cabdca8b242d397b00a90d55ef9ec
I20240501 19:59:42.990047 25719 ts_tablet_manager.cc:1398] T e659b732ef82488eac87492a463bd702 P 016da4070e314b2786e8614d8a954ecd: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20240501 19:59:42.990347 25736 raft_consensus.cc:2781] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:59:42.990806 25736 raft_consensus.cc:686] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [term 1 LEADER]: Becoming Leader. State: Replica: 5d2ab23455b64a75a08d71d0db37be10, State: Running, Role: LEADER
I20240501 19:59:42.991624 25736 consensus_queue.cc:237] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:42.992560 25720 ts_tablet_manager.cc:1398] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20240501 19:59:42.993538 25720 tablet_bootstrap.cc:492] T 38c3d8dabd6648ff9a4262c784b88c80 P 5d2ab23455b64a75a08d71d0db37be10: Bootstrap starting.
I20240501 19:59:42.993384 25721 tablet_bootstrap.cc:492] T 3a18557702154a5a91ff0d36e4df951d P 7f6cabdca8b242d397b00a90d55ef9ec: No bootstrap required, opened a new log
I20240501 19:59:42.994392 25721 ts_tablet_manager.cc:1367] T 3a18557702154a5a91ff0d36e4df951d P 7f6cabdca8b242d397b00a90d55ef9ec: Time spent bootstrapping tablet: real 0.041s user 0.025s sys 0.015s
I20240501 19:59:42.997387 25721 raft_consensus.cc:348] T 3a18557702154a5a91ff0d36e4df951d P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:42.997936 25719 tablet_bootstrap.cc:492] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd: Bootstrap starting.
I20240501 19:59:42.998281 25721 raft_consensus.cc:374] T 3a18557702154a5a91ff0d36e4df951d P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:42.998670 25721 raft_consensus.cc:729] T 3a18557702154a5a91ff0d36e4df951d P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6cabdca8b242d397b00a90d55ef9ec, State: Initialized, Role: FOLLOWER
I20240501 19:59:42.998935 25720 tablet_bootstrap.cc:654] T 38c3d8dabd6648ff9a4262c784b88c80 P 5d2ab23455b64a75a08d71d0db37be10: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:42.999588 25721 consensus_queue.cc:260] T 3a18557702154a5a91ff0d36e4df951d P 7f6cabdca8b242d397b00a90d55ef9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.003283 25721 ts_tablet_manager.cc:1398] T 3a18557702154a5a91ff0d36e4df951d P 7f6cabdca8b242d397b00a90d55ef9ec: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240501 19:59:43.004226 25721 tablet_bootstrap.cc:492] T 47f6e1c34803471289c79387682db511 P 7f6cabdca8b242d397b00a90d55ef9ec: Bootstrap starting.
I20240501 19:59:43.005789 25719 tablet_bootstrap.cc:654] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:43.010697 25721 tablet_bootstrap.cc:654] T 47f6e1c34803471289c79387682db511 P 7f6cabdca8b242d397b00a90d55ef9ec: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:43.013581 25423 catalog_manager.cc:5484] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d2ab23455b64a75a08d71d0db37be10 (127.24.69.130). New cstate: current_term: 1 leader_uuid: "5d2ab23455b64a75a08d71d0db37be10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } health_report { overall_health: UNKNOWN } } }
I20240501 19:59:43.019359 25736 raft_consensus.cc:482] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:59:43.019870 25719 tablet_bootstrap.cc:492] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd: No bootstrap required, opened a new log
I20240501 19:59:43.019795 25736 raft_consensus.cc:504] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.020567 25719 ts_tablet_manager.cc:1367] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.003s
I20240501 19:59:43.024070 25674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a18557702154a5a91ff0d36e4df951d" candidate_uuid: "5d2ab23455b64a75a08d71d0db37be10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016da4070e314b2786e8614d8a954ecd" is_pre_election: true
I20240501 19:59:43.023859 25719 raft_consensus.cc:348] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } }
I20240501 19:59:43.024767 25674 raft_consensus.cc:2454] T 3a18557702154a5a91ff0d36e4df951d P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d2ab23455b64a75a08d71d0db37be10 in term 0.
I20240501 19:59:43.024914 25719 raft_consensus.cc:374] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:43.025373 25719 raft_consensus.cc:729] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 016da4070e314b2786e8614d8a954ecd, State: Initialized, Role: FOLLOWER
I20240501 19:59:43.025734 25525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a18557702154a5a91ff0d36e4df951d" candidate_uuid: "5d2ab23455b64a75a08d71d0db37be10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" is_pre_election: true
I20240501 19:59:43.026340 25525 raft_consensus.cc:2454] T 3a18557702154a5a91ff0d36e4df951d P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d2ab23455b64a75a08d71d0db37be10 in term 0.
I20240501 19:59:43.026226 25719 consensus_queue.cc:260] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } }
I20240501 19:59:43.027555 25563 leader_election.cc:302] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d2ab23455b64a75a08d71d0db37be10, 7f6cabdca8b242d397b00a90d55ef9ec; no voters:
I20240501 19:59:43.028378 25726 raft_consensus.cc:2781] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:59:43.028779 25726 raft_consensus.cc:482] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:59:43.029119 25726 raft_consensus.cc:3037] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:43.029127 25719 ts_tablet_manager.cc:1398] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240501 19:59:43.030429 25719 tablet_bootstrap.cc:492] T ba5ada435b794cc194e3cbff0d994915 P 016da4070e314b2786e8614d8a954ecd: Bootstrap starting.
I20240501 19:59:43.033919 25736 leader_election.cc:288] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129:39987), 016da4070e314b2786e8614d8a954ecd (127.24.69.131:42353)
I20240501 19:59:43.036051 25726 raft_consensus.cc:504] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.037068 25719 tablet_bootstrap.cc:654] T ba5ada435b794cc194e3cbff0d994915 P 016da4070e314b2786e8614d8a954ecd: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:43.037566 25720 tablet_bootstrap.cc:492] T 38c3d8dabd6648ff9a4262c784b88c80 P 5d2ab23455b64a75a08d71d0db37be10: No bootstrap required, opened a new log
I20240501 19:59:43.038085 25720 ts_tablet_manager.cc:1367] T 38c3d8dabd6648ff9a4262c784b88c80 P 5d2ab23455b64a75a08d71d0db37be10: Time spent bootstrapping tablet: real 0.045s user 0.016s sys 0.004s
I20240501 19:59:43.040565 25720 raft_consensus.cc:348] T 38c3d8dabd6648ff9a4262c784b88c80 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.041340 25720 raft_consensus.cc:374] T 38c3d8dabd6648ff9a4262c784b88c80 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:43.041803 25720 raft_consensus.cc:729] T 38c3d8dabd6648ff9a4262c784b88c80 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d2ab23455b64a75a08d71d0db37be10, State: Initialized, Role: FOLLOWER
I20240501 19:59:43.043931 25720 consensus_queue.cc:260] T 38c3d8dabd6648ff9a4262c784b88c80 P 5d2ab23455b64a75a08d71d0db37be10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.043485 25421 catalog_manager.cc:5484] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d2ab23455b64a75a08d71d0db37be10 (127.24.69.130). New cstate: current_term: 1 leader_uuid: "5d2ab23455b64a75a08d71d0db37be10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } health_report { overall_health: UNKNOWN } } }
I20240501 19:59:43.045159 25721 tablet_bootstrap.cc:492] T 47f6e1c34803471289c79387682db511 P 7f6cabdca8b242d397b00a90d55ef9ec: No bootstrap required, opened a new log
I20240501 19:59:43.045133 25525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a18557702154a5a91ff0d36e4df951d" candidate_uuid: "5d2ab23455b64a75a08d71d0db37be10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6cabdca8b242d397b00a90d55ef9ec"
I20240501 19:59:43.045791 25721 ts_tablet_manager.cc:1367] T 47f6e1c34803471289c79387682db511 P 7f6cabdca8b242d397b00a90d55ef9ec: Time spent bootstrapping tablet: real 0.042s user 0.023s sys 0.006s
I20240501 19:59:43.045904 25525 raft_consensus.cc:3037] T 3a18557702154a5a91ff0d36e4df951d P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:43.047258 25720 ts_tablet_manager.cc:1398] T 38c3d8dabd6648ff9a4262c784b88c80 P 5d2ab23455b64a75a08d71d0db37be10: Time spent starting tablet: real 0.009s user 0.003s sys 0.003s
I20240501 19:59:43.048243 25720 tablet_bootstrap.cc:492] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10: Bootstrap starting.
I20240501 19:59:43.048775 25721 raft_consensus.cc:348] T 47f6e1c34803471289c79387682db511 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.049486 25721 raft_consensus.cc:374] T 47f6e1c34803471289c79387682db511 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:43.049772 25721 raft_consensus.cc:729] T 47f6e1c34803471289c79387682db511 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6cabdca8b242d397b00a90d55ef9ec, State: Initialized, Role: FOLLOWER
I20240501 19:59:43.050437 25721 consensus_queue.cc:260] T 47f6e1c34803471289c79387682db511 P 7f6cabdca8b242d397b00a90d55ef9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.052659 25721 ts_tablet_manager.cc:1398] T 47f6e1c34803471289c79387682db511 P 7f6cabdca8b242d397b00a90d55ef9ec: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20240501 19:59:43.052713 25525 raft_consensus.cc:2454] T 3a18557702154a5a91ff0d36e4df951d P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d2ab23455b64a75a08d71d0db37be10 in term 1.
I20240501 19:59:43.053691 25721 tablet_bootstrap.cc:492] T 7320413158f34e7d80d98dd88b562af7 P 7f6cabdca8b242d397b00a90d55ef9ec: Bootstrap starting.
I20240501 19:59:43.055511 25563 leader_election.cc:302] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d2ab23455b64a75a08d71d0db37be10, 7f6cabdca8b242d397b00a90d55ef9ec; no voters:
I20240501 19:59:43.056701 25736 raft_consensus.cc:2781] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:59:43.057147 25736 raft_consensus.cc:686] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10 [term 1 LEADER]: Becoming Leader. State: Replica: 5d2ab23455b64a75a08d71d0db37be10, State: Running, Role: LEADER
I20240501 19:59:43.059401 25674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a18557702154a5a91ff0d36e4df951d" candidate_uuid: "5d2ab23455b64a75a08d71d0db37be10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016da4070e314b2786e8614d8a954ecd"
I20240501 19:59:43.059995 25674 raft_consensus.cc:3037] T 3a18557702154a5a91ff0d36e4df951d P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:43.060730 25719 tablet_bootstrap.cc:492] T ba5ada435b794cc194e3cbff0d994915 P 016da4070e314b2786e8614d8a954ecd: No bootstrap required, opened a new log
I20240501 19:59:43.061173 25719 ts_tablet_manager.cc:1367] T ba5ada435b794cc194e3cbff0d994915 P 016da4070e314b2786e8614d8a954ecd: Time spent bootstrapping tablet: real 0.031s user 0.014s sys 0.001s
I20240501 19:59:43.058493 25736 consensus_queue.cc:237] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.061539 25728 raft_consensus.cc:482] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:59:43.062016 25728 raft_consensus.cc:504] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.063465 25720 tablet_bootstrap.cc:654] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:43.065207 25728 leader_election.cc:288] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d2ab23455b64a75a08d71d0db37be10 (127.24.69.130:33157), 016da4070e314b2786e8614d8a954ecd (127.24.69.131:42353)
I20240501 19:59:43.061286 25726 leader_election.cc:288] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 election: Requested vote from peers 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129:39987), 016da4070e314b2786e8614d8a954ecd (127.24.69.131:42353)
I20240501 19:59:43.067481 25721 tablet_bootstrap.cc:654] T 7320413158f34e7d80d98dd88b562af7 P 7f6cabdca8b242d397b00a90d55ef9ec: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:43.070509 25719 raft_consensus.cc:348] T ba5ada435b794cc194e3cbff0d994915 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.071260 25719 raft_consensus.cc:374] T ba5ada435b794cc194e3cbff0d994915 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:43.071543 25719 raft_consensus.cc:729] T ba5ada435b794cc194e3cbff0d994915 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 016da4070e314b2786e8614d8a954ecd, State: Initialized, Role: FOLLOWER
I20240501 19:59:43.072221 25719 consensus_queue.cc:260] T ba5ada435b794cc194e3cbff0d994915 P 016da4070e314b2786e8614d8a954ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.073503 25674 raft_consensus.cc:2454] T 3a18557702154a5a91ff0d36e4df951d P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d2ab23455b64a75a08d71d0db37be10 in term 1.
I20240501 19:59:43.077152 25719 ts_tablet_manager.cc:1398] T ba5ada435b794cc194e3cbff0d994915 P 016da4070e314b2786e8614d8a954ecd: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20240501 19:59:43.080730 25724 raft_consensus.cc:482] T a528234b11234a6a8516d9365b49d8db P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:59:43.081342 25724 raft_consensus.cc:504] T a528234b11234a6a8516d9365b49d8db P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.085603 25719 tablet_bootstrap.cc:492] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd: Bootstrap starting.
I20240501 19:59:43.086726 25724 leader_election.cc:288] T a528234b11234a6a8516d9365b49d8db P 016da4070e314b2786e8614d8a954ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129:39987), 5d2ab23455b64a75a08d71d0db37be10 (127.24.69.130:33157)
I20240501 19:59:43.102921 25736 raft_consensus.cc:482] T 38c3d8dabd6648ff9a4262c784b88c80 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:59:43.109301 25736 raft_consensus.cc:504] T 38c3d8dabd6648ff9a4262c784b88c80 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.114109 25525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c3d8dabd6648ff9a4262c784b88c80" candidate_uuid: "5d2ab23455b64a75a08d71d0db37be10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" is_pre_election: true
W20240501 19:59:43.115722 25563 leader_election.cc:341] T 38c3d8dabd6648ff9a4262c784b88c80 P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129:39987): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:59:43.116045 25422 catalog_manager.cc:5484] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d2ab23455b64a75a08d71d0db37be10 (127.24.69.130). New cstate: current_term: 1 leader_uuid: "5d2ab23455b64a75a08d71d0db37be10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } health_report { overall_health: UNKNOWN } } }
I20240501 19:59:43.117789 25674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c3d8dabd6648ff9a4262c784b88c80" candidate_uuid: "5d2ab23455b64a75a08d71d0db37be10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016da4070e314b2786e8614d8a954ecd" is_pre_election: true
I20240501 19:59:43.115985 25736 leader_election.cc:288] T 38c3d8dabd6648ff9a4262c784b88c80 P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129:39987), 016da4070e314b2786e8614d8a954ecd (127.24.69.131:42353)
W20240501 19:59:43.119657 25564 leader_election.cc:341] T 38c3d8dabd6648ff9a4262c784b88c80 P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 016da4070e314b2786e8614d8a954ecd (127.24.69.131:42353): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:59:43.123407 25721 tablet_bootstrap.cc:492] T 7320413158f34e7d80d98dd88b562af7 P 7f6cabdca8b242d397b00a90d55ef9ec: No bootstrap required, opened a new log
I20240501 19:59:43.125296 25721 ts_tablet_manager.cc:1367] T 7320413158f34e7d80d98dd88b562af7 P 7f6cabdca8b242d397b00a90d55ef9ec: Time spent bootstrapping tablet: real 0.072s user 0.008s sys 0.018s
I20240501 19:59:43.123706 25564 leader_election.cc:302] T 38c3d8dabd6648ff9a4262c784b88c80 P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d2ab23455b64a75a08d71d0db37be10; no voters: 016da4070e314b2786e8614d8a954ecd, 7f6cabdca8b242d397b00a90d55ef9ec
I20240501 19:59:43.127539 25724 raft_consensus.cc:482] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:59:43.127972 25724 raft_consensus.cc:504] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.128432 25736 raft_consensus.cc:2726] T 38c3d8dabd6648ff9a4262c784b88c80 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:59:43.130532 25724 leader_election.cc:288] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129:39987), 5d2ab23455b64a75a08d71d0db37be10 (127.24.69.130:33157)
I20240501 19:59:43.132493 25719 tablet_bootstrap.cc:654] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:43.134598 25600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba5ada435b794cc194e3cbff0d994915" candidate_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d2ab23455b64a75a08d71d0db37be10" is_pre_election: true
W20240501 19:59:43.136276 25490 leader_election.cc:341] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d2ab23455b64a75a08d71d0db37be10 (127.24.69.130:33157): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:59:43.138401 25720 tablet_bootstrap.cc:492] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10: No bootstrap required, opened a new log
I20240501 19:59:43.138231 25721 raft_consensus.cc:348] T 7320413158f34e7d80d98dd88b562af7 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.139295 25721 raft_consensus.cc:374] T 7320413158f34e7d80d98dd88b562af7 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:43.139297 25720 ts_tablet_manager.cc:1367] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10: Time spent bootstrapping tablet: real 0.091s user 0.004s sys 0.014s
I20240501 19:59:43.139653 25721 raft_consensus.cc:729] T 7320413158f34e7d80d98dd88b562af7 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6cabdca8b242d397b00a90d55ef9ec, State: Initialized, Role: FOLLOWER
I20240501 19:59:43.141247 25721 consensus_queue.cc:260] T 7320413158f34e7d80d98dd88b562af7 P 7f6cabdca8b242d397b00a90d55ef9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.143538 25674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba5ada435b794cc194e3cbff0d994915" candidate_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016da4070e314b2786e8614d8a954ecd" is_pre_election: true
I20240501 19:59:43.146070 25674 raft_consensus.cc:2454] T ba5ada435b794cc194e3cbff0d994915 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f6cabdca8b242d397b00a90d55ef9ec in term 0.
I20240501 19:59:43.145476 25720 raft_consensus.cc:348] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.146728 25720 raft_consensus.cc:374] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:43.147177 25720 raft_consensus.cc:729] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d2ab23455b64a75a08d71d0db37be10, State: Initialized, Role: FOLLOWER
I20240501 19:59:43.148098 25720 consensus_queue.cc:260] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.150161 25489 leader_election.cc:302] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 016da4070e314b2786e8614d8a954ecd, 7f6cabdca8b242d397b00a90d55ef9ec; no voters: 5d2ab23455b64a75a08d71d0db37be10
I20240501 19:59:43.151161 25728 raft_consensus.cc:2781] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:59:43.151597 25728 raft_consensus.cc:482] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:59:43.151993 25728 raft_consensus.cc:3037] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:43.152897 25720 ts_tablet_manager.cc:1398] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10: Time spent starting tablet: real 0.012s user 0.004s sys 0.004s
I20240501 19:59:43.154739 25720 tablet_bootstrap.cc:492] T ba5ada435b794cc194e3cbff0d994915 P 5d2ab23455b64a75a08d71d0db37be10: Bootstrap starting.
I20240501 19:59:43.158771 25721 ts_tablet_manager.cc:1398] T 7320413158f34e7d80d98dd88b562af7 P 7f6cabdca8b242d397b00a90d55ef9ec: Time spent starting tablet: real 0.032s user 0.003s sys 0.004s
I20240501 19:59:43.158586 25599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7320413158f34e7d80d98dd88b562af7" candidate_uuid: "016da4070e314b2786e8614d8a954ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d2ab23455b64a75a08d71d0db37be10" is_pre_election: true
I20240501 19:59:43.158939 25600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a528234b11234a6a8516d9365b49d8db" candidate_uuid: "016da4070e314b2786e8614d8a954ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d2ab23455b64a75a08d71d0db37be10" is_pre_election: true
I20240501 19:59:43.159729 25721 tablet_bootstrap.cc:492] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec: Bootstrap starting.
W20240501 19:59:43.161183 25639 leader_election.cc:341] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d2ab23455b64a75a08d71d0db37be10 (127.24.69.130:33157): Illegal state: must be running to vote when last-logged opid is not known
W20240501 19:59:43.161974 25639 leader_election.cc:341] T a528234b11234a6a8516d9365b49d8db P 016da4070e314b2786e8614d8a954ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d2ab23455b64a75a08d71d0db37be10 (127.24.69.130:33157): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:59:43.164431 25728 raft_consensus.cc:504] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.167245 25721 tablet_bootstrap.cc:654] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:43.168522 25525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a528234b11234a6a8516d9365b49d8db" candidate_uuid: "016da4070e314b2786e8614d8a954ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" is_pre_election: true
I20240501 19:59:43.169827 25720 tablet_bootstrap.cc:654] T ba5ada435b794cc194e3cbff0d994915 P 5d2ab23455b64a75a08d71d0db37be10: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:43.170428 25524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7320413158f34e7d80d98dd88b562af7" candidate_uuid: "016da4070e314b2786e8614d8a954ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" is_pre_election: true
I20240501 19:59:43.172181 25476 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:59:43.172421 25524 raft_consensus.cc:2454] T 7320413158f34e7d80d98dd88b562af7 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 016da4070e314b2786e8614d8a954ecd in term 0.
W20240501 19:59:43.172219 25637 leader_election.cc:341] T a528234b11234a6a8516d9365b49d8db P 016da4070e314b2786e8614d8a954ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129:39987): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:59:43.174177 25637 leader_election.cc:302] T a528234b11234a6a8516d9365b49d8db P 016da4070e314b2786e8614d8a954ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 016da4070e314b2786e8614d8a954ecd; no voters: 5d2ab23455b64a75a08d71d0db37be10, 7f6cabdca8b242d397b00a90d55ef9ec
I20240501 19:59:43.175242 25724 raft_consensus.cc:2726] T a528234b11234a6a8516d9365b49d8db P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 19:59:43.175056 25471 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 19:59:43.175673 25637 leader_election.cc:302] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 016da4070e314b2786e8614d8a954ecd, 7f6cabdca8b242d397b00a90d55ef9ec; no voters: 5d2ab23455b64a75a08d71d0db37be10
I20240501 19:59:43.176506 25724 raft_consensus.cc:2781] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:59:43.176486 25600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba5ada435b794cc194e3cbff0d994915" candidate_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d2ab23455b64a75a08d71d0db37be10"
I20240501 19:59:43.177070 25724 raft_consensus.cc:482] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:59:43.177428 25724 raft_consensus.cc:3037] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Advancing to term 1
W20240501 19:59:43.177946 25490 leader_election.cc:341] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5d2ab23455b64a75a08d71d0db37be10 (127.24.69.130:33157): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:59:43.178746 25728 leader_election.cc:288] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec [CANDIDATE]: Term 1 election: Requested vote from peers 5d2ab23455b64a75a08d71d0db37be10 (127.24.69.130:33157), 016da4070e314b2786e8614d8a954ecd (127.24.69.131:42353)
I20240501 19:59:43.179306 25719 tablet_bootstrap.cc:492] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd: No bootstrap required, opened a new log
I20240501 19:59:43.179711 25674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba5ada435b794cc194e3cbff0d994915" candidate_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016da4070e314b2786e8614d8a954ecd"
I20240501 19:59:43.179816 25719 ts_tablet_manager.cc:1367] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd: Time spent bootstrapping tablet: real 0.099s user 0.016s sys 0.012s
I20240501 19:59:43.180182 25674 raft_consensus.cc:3037] T ba5ada435b794cc194e3cbff0d994915 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:43.181084 25476 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:59:43.185281 25721 tablet_bootstrap.cc:492] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec: No bootstrap required, opened a new log
I20240501 19:59:43.185671 25721 ts_tablet_manager.cc:1367] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.001s
I20240501 19:59:43.185706 25724 raft_consensus.cc:504] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.187088 25674 raft_consensus.cc:2454] T ba5ada435b794cc194e3cbff0d994915 P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f6cabdca8b242d397b00a90d55ef9ec in term 1.
I20240501 19:59:43.188807 25489 leader_election.cc:302] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 016da4070e314b2786e8614d8a954ecd, 7f6cabdca8b242d397b00a90d55ef9ec; no voters: 5d2ab23455b64a75a08d71d0db37be10
I20240501 19:59:43.188843 25524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7320413158f34e7d80d98dd88b562af7" candidate_uuid: "016da4070e314b2786e8614d8a954ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6cabdca8b242d397b00a90d55ef9ec"
I20240501 19:59:43.188864 25721 raft_consensus.cc:348] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.189703 25524 raft_consensus.cc:3037] T 7320413158f34e7d80d98dd88b562af7 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:43.189860 25721 raft_consensus.cc:374] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:43.190263 25721 raft_consensus.cc:729] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6cabdca8b242d397b00a90d55ef9ec, State: Initialized, Role: FOLLOWER
I20240501 19:59:43.190141 25728 raft_consensus.cc:2781] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:59:43.190435 25719 raft_consensus.cc:348] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.191287 25719 raft_consensus.cc:374] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:43.191634 25719 raft_consensus.cc:729] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 016da4070e314b2786e8614d8a954ecd, State: Initialized, Role: FOLLOWER
I20240501 19:59:43.191174 25721 consensus_queue.cc:260] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.193392 25600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7320413158f34e7d80d98dd88b562af7" candidate_uuid: "016da4070e314b2786e8614d8a954ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d2ab23455b64a75a08d71d0db37be10"
I20240501 19:59:43.193665 25724 leader_election.cc:288] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd [CANDIDATE]: Term 1 election: Requested vote from peers 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129:39987), 5d2ab23455b64a75a08d71d0db37be10 (127.24.69.130:33157)
I20240501 19:59:43.194700 25720 tablet_bootstrap.cc:492] T ba5ada435b794cc194e3cbff0d994915 P 5d2ab23455b64a75a08d71d0db37be10: No bootstrap required, opened a new log
W20240501 19:59:43.194815 25639 leader_election.cc:341] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5d2ab23455b64a75a08d71d0db37be10 (127.24.69.130:33157): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:59:43.195240 25720 ts_tablet_manager.cc:1367] T ba5ada435b794cc194e3cbff0d994915 P 5d2ab23455b64a75a08d71d0db37be10: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.003s
I20240501 19:59:43.192452 25719 consensus_queue.cc:260] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.197446 25719 ts_tablet_manager.cc:1398] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd: Time spent starting tablet: real 0.017s user 0.003s sys 0.002s
I20240501 19:59:43.198051 25524 raft_consensus.cc:2454] T 7320413158f34e7d80d98dd88b562af7 P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 016da4070e314b2786e8614d8a954ecd in term 1.
I20240501 19:59:43.197984 25720 raft_consensus.cc:348] T ba5ada435b794cc194e3cbff0d994915 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.198669 25720 raft_consensus.cc:374] T ba5ada435b794cc194e3cbff0d994915 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:43.198966 25720 raft_consensus.cc:729] T ba5ada435b794cc194e3cbff0d994915 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d2ab23455b64a75a08d71d0db37be10, State: Initialized, Role: FOLLOWER
I20240501 19:59:43.199391 25637 leader_election.cc:302] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 016da4070e314b2786e8614d8a954ecd, 7f6cabdca8b242d397b00a90d55ef9ec; no voters: 5d2ab23455b64a75a08d71d0db37be10
I20240501 19:59:43.199693 25720 consensus_queue.cc:260] T ba5ada435b794cc194e3cbff0d994915 P 5d2ab23455b64a75a08d71d0db37be10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.200858 25724 raft_consensus.cc:2781] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:59:43.202580 25728 raft_consensus.cc:686] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 LEADER]: Becoming Leader. State: Replica: 7f6cabdca8b242d397b00a90d55ef9ec, State: Running, Role: LEADER
I20240501 19:59:43.203387 25728 consensus_queue.cc:237] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.208426 25724 raft_consensus.cc:686] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd [term 1 LEADER]: Becoming Leader. State: Replica: 016da4070e314b2786e8614d8a954ecd, State: Running, Role: LEADER
I20240501 19:59:43.209527 25720 ts_tablet_manager.cc:1398] T ba5ada435b794cc194e3cbff0d994915 P 5d2ab23455b64a75a08d71d0db37be10: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20240501 19:59:43.209923 25724 consensus_queue.cc:237] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.212291 25720 tablet_bootstrap.cc:492] T 7320413158f34e7d80d98dd88b562af7 P 5d2ab23455b64a75a08d71d0db37be10: Bootstrap starting.
I20240501 19:59:43.220590 25721 ts_tablet_manager.cc:1398] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec: Time spent starting tablet: real 0.035s user 0.015s sys 0.001s
I20240501 19:59:43.221973 25721 tablet_bootstrap.cc:492] T 38c3d8dabd6648ff9a4262c784b88c80 P 7f6cabdca8b242d397b00a90d55ef9ec: Bootstrap starting.
I20240501 19:59:43.225354 25759 raft_consensus.cc:482] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:59:43.226227 25759 raft_consensus.cc:504] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.229687 25524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c3d8dabd6648ff9a4262c784b88c80" candidate_uuid: "016da4070e314b2786e8614d8a954ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" is_pre_election: true
I20240501 19:59:43.230290 25721 tablet_bootstrap.cc:654] T 38c3d8dabd6648ff9a4262c784b88c80 P 7f6cabdca8b242d397b00a90d55ef9ec: Neither blocks nor log segments found. Creating new log.
W20240501 19:59:43.231063 25637 leader_election.cc:341] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129:39987): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:59:43.233090 25759 leader_election.cc:288] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129:39987), 5d2ab23455b64a75a08d71d0db37be10 (127.24.69.130:33157)
I20240501 19:59:43.234105 25600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c3d8dabd6648ff9a4262c784b88c80" candidate_uuid: "016da4070e314b2786e8614d8a954ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d2ab23455b64a75a08d71d0db37be10" is_pre_election: true
I20240501 19:59:43.234781 25600 raft_consensus.cc:2454] T 38c3d8dabd6648ff9a4262c784b88c80 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 016da4070e314b2786e8614d8a954ecd in term 0.
I20240501 19:59:43.236028 25639 leader_election.cc:302] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 016da4070e314b2786e8614d8a954ecd, 5d2ab23455b64a75a08d71d0db37be10; no voters: 7f6cabdca8b242d397b00a90d55ef9ec
I20240501 19:59:43.237200 25759 raft_consensus.cc:2781] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:59:43.237627 25759 raft_consensus.cc:482] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:59:43.238045 25759 raft_consensus.cc:3037] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:43.237946 25422 catalog_manager.cc:5484] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd reported cstate change: term changed from 0 to 1, leader changed from <none> to 016da4070e314b2786e8614d8a954ecd (127.24.69.131). New cstate: current_term: 1 leader_uuid: "016da4070e314b2786e8614d8a954ecd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } health_report { overall_health: HEALTHY } } }
I20240501 19:59:43.237948 25423 catalog_manager.cc:5484] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129). New cstate: current_term: 1 leader_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } health_report { overall_health: UNKNOWN } } }
I20240501 19:59:43.244858 25720 tablet_bootstrap.cc:654] T 7320413158f34e7d80d98dd88b562af7 P 5d2ab23455b64a75a08d71d0db37be10: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:43.246093 25759 raft_consensus.cc:504] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.248857 25524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c3d8dabd6648ff9a4262c784b88c80" candidate_uuid: "016da4070e314b2786e8614d8a954ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6cabdca8b242d397b00a90d55ef9ec"
I20240501 19:59:43.249336 25600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c3d8dabd6648ff9a4262c784b88c80" candidate_uuid: "016da4070e314b2786e8614d8a954ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d2ab23455b64a75a08d71d0db37be10"
I20240501 19:59:43.250104 25600 raft_consensus.cc:3037] T 38c3d8dabd6648ff9a4262c784b88c80 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Advancing to term 1
W20240501 19:59:43.250159 25637 leader_election.cc:341] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129:39987): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:59:43.252199 25759 leader_election.cc:288] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd [CANDIDATE]: Term 1 election: Requested vote from peers 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129:39987), 5d2ab23455b64a75a08d71d0db37be10 (127.24.69.130:33157)
I20240501 19:59:43.255101 25736 raft_consensus.cc:482] T ba5ada435b794cc194e3cbff0d994915 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:59:43.255739 25736 raft_consensus.cc:504] T ba5ada435b794cc194e3cbff0d994915 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.257784 25736 leader_election.cc:288] T ba5ada435b794cc194e3cbff0d994915 P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129:39987), 016da4070e314b2786e8614d8a954ecd (127.24.69.131:42353)
I20240501 19:59:43.258793 25721 tablet_bootstrap.cc:492] T 38c3d8dabd6648ff9a4262c784b88c80 P 7f6cabdca8b242d397b00a90d55ef9ec: No bootstrap required, opened a new log
I20240501 19:59:43.259234 25721 ts_tablet_manager.cc:1367] T 38c3d8dabd6648ff9a4262c784b88c80 P 7f6cabdca8b242d397b00a90d55ef9ec: Time spent bootstrapping tablet: real 0.038s user 0.010s sys 0.001s
I20240501 19:59:43.259145 25674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba5ada435b794cc194e3cbff0d994915" candidate_uuid: "5d2ab23455b64a75a08d71d0db37be10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016da4070e314b2786e8614d8a954ecd" is_pre_election: true
I20240501 19:59:43.259723 25524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba5ada435b794cc194e3cbff0d994915" candidate_uuid: "5d2ab23455b64a75a08d71d0db37be10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" is_pre_election: true
I20240501 19:59:43.262053 25721 raft_consensus.cc:348] T 38c3d8dabd6648ff9a4262c784b88c80 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.262777 25721 raft_consensus.cc:374] T 38c3d8dabd6648ff9a4262c784b88c80 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:43.263100 25721 raft_consensus.cc:729] T 38c3d8dabd6648ff9a4262c784b88c80 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6cabdca8b242d397b00a90d55ef9ec, State: Initialized, Role: FOLLOWER
I20240501 19:59:43.263921 25721 consensus_queue.cc:260] T 38c3d8dabd6648ff9a4262c784b88c80 P 7f6cabdca8b242d397b00a90d55ef9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.266067 25674 raft_consensus.cc:2379] T ba5ada435b794cc194e3cbff0d994915 P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d2ab23455b64a75a08d71d0db37be10 in current term 1: Already voted for candidate 7f6cabdca8b242d397b00a90d55ef9ec in this term.
I20240501 19:59:43.266558 25721 ts_tablet_manager.cc:1398] T 38c3d8dabd6648ff9a4262c784b88c80 P 7f6cabdca8b242d397b00a90d55ef9ec: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240501 19:59:43.267863 25564 leader_election.cc:302] T ba5ada435b794cc194e3cbff0d994915 P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d2ab23455b64a75a08d71d0db37be10; no voters: 016da4070e314b2786e8614d8a954ecd, 7f6cabdca8b242d397b00a90d55ef9ec
I20240501 19:59:43.269394 25736 raft_consensus.cc:3037] T ba5ada435b794cc194e3cbff0d994915 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:43.267519 25721 tablet_bootstrap.cc:492] T e659b732ef82488eac87492a463bd702 P 7f6cabdca8b242d397b00a90d55ef9ec: Bootstrap starting.
I20240501 19:59:43.275570 25600 raft_consensus.cc:2454] T 38c3d8dabd6648ff9a4262c784b88c80 P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 016da4070e314b2786e8614d8a954ecd in term 1.
I20240501 19:59:43.276783 25639 leader_election.cc:302] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 016da4070e314b2786e8614d8a954ecd, 5d2ab23455b64a75a08d71d0db37be10; no voters: 7f6cabdca8b242d397b00a90d55ef9ec
I20240501 19:59:43.277146 25721 tablet_bootstrap.cc:654] T e659b732ef82488eac87492a463bd702 P 7f6cabdca8b242d397b00a90d55ef9ec: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:43.278625 25736 raft_consensus.cc:2726] T ba5ada435b794cc194e3cbff0d994915 P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:59:43.279088 25759 raft_consensus.cc:2781] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:59:43.280112 25720 tablet_bootstrap.cc:492] T 7320413158f34e7d80d98dd88b562af7 P 5d2ab23455b64a75a08d71d0db37be10: No bootstrap required, opened a new log
I20240501 19:59:43.280373 25759 raft_consensus.cc:686] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd [term 1 LEADER]: Becoming Leader. State: Replica: 016da4070e314b2786e8614d8a954ecd, State: Running, Role: LEADER
I20240501 19:59:43.281647 25720 ts_tablet_manager.cc:1367] T 7320413158f34e7d80d98dd88b562af7 P 5d2ab23455b64a75a08d71d0db37be10: Time spent bootstrapping tablet: real 0.070s user 0.020s sys 0.022s
I20240501 19:59:43.281591 25759 consensus_queue.cc:237] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.283540 25720 raft_consensus.cc:348] T 7320413158f34e7d80d98dd88b562af7 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.284621 25720 raft_consensus.cc:374] T 7320413158f34e7d80d98dd88b562af7 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:43.284960 25720 raft_consensus.cc:729] T 7320413158f34e7d80d98dd88b562af7 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d2ab23455b64a75a08d71d0db37be10, State: Initialized, Role: FOLLOWER
I20240501 19:59:43.285714 25720 consensus_queue.cc:260] T 7320413158f34e7d80d98dd88b562af7 P 5d2ab23455b64a75a08d71d0db37be10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.286588 25721 tablet_bootstrap.cc:492] T e659b732ef82488eac87492a463bd702 P 7f6cabdca8b242d397b00a90d55ef9ec: No bootstrap required, opened a new log
I20240501 19:59:43.287034 25721 ts_tablet_manager.cc:1367] T e659b732ef82488eac87492a463bd702 P 7f6cabdca8b242d397b00a90d55ef9ec: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.005s
I20240501 19:59:43.287942 25720 ts_tablet_manager.cc:1398] T 7320413158f34e7d80d98dd88b562af7 P 5d2ab23455b64a75a08d71d0db37be10: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240501 19:59:43.289899 25720 tablet_bootstrap.cc:492] T a528234b11234a6a8516d9365b49d8db P 5d2ab23455b64a75a08d71d0db37be10: Bootstrap starting.
I20240501 19:59:43.289556 25721 raft_consensus.cc:348] T e659b732ef82488eac87492a463bd702 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.290251 25721 raft_consensus.cc:374] T e659b732ef82488eac87492a463bd702 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:43.290514 25721 raft_consensus.cc:729] T e659b732ef82488eac87492a463bd702 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6cabdca8b242d397b00a90d55ef9ec, State: Initialized, Role: FOLLOWER
I20240501 19:59:43.291147 25721 consensus_queue.cc:260] T e659b732ef82488eac87492a463bd702 P 7f6cabdca8b242d397b00a90d55ef9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.291134 25422 catalog_manager.cc:5484] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd reported cstate change: term changed from 0 to 1, leader changed from <none> to 016da4070e314b2786e8614d8a954ecd (127.24.69.131). New cstate: current_term: 1 leader_uuid: "016da4070e314b2786e8614d8a954ecd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } health_report { overall_health: HEALTHY } } }
I20240501 19:59:43.295773 25721 ts_tablet_manager.cc:1398] T e659b732ef82488eac87492a463bd702 P 7f6cabdca8b242d397b00a90d55ef9ec: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20240501 19:59:43.298902 25720 tablet_bootstrap.cc:654] T a528234b11234a6a8516d9365b49d8db P 5d2ab23455b64a75a08d71d0db37be10: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:43.303712 25720 tablet_bootstrap.cc:492] T a528234b11234a6a8516d9365b49d8db P 5d2ab23455b64a75a08d71d0db37be10: No bootstrap required, opened a new log
I20240501 19:59:43.304050 25720 ts_tablet_manager.cc:1367] T a528234b11234a6a8516d9365b49d8db P 5d2ab23455b64a75a08d71d0db37be10: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.001s
I20240501 19:59:43.305961 25720 raft_consensus.cc:348] T a528234b11234a6a8516d9365b49d8db P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.306427 25720 raft_consensus.cc:374] T a528234b11234a6a8516d9365b49d8db P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:43.306615 25720 raft_consensus.cc:729] T a528234b11234a6a8516d9365b49d8db P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d2ab23455b64a75a08d71d0db37be10, State: Initialized, Role: FOLLOWER
I20240501 19:59:43.307035 25720 consensus_queue.cc:260] T a528234b11234a6a8516d9365b49d8db P 5d2ab23455b64a75a08d71d0db37be10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.308493 25720 ts_tablet_manager.cc:1398] T a528234b11234a6a8516d9365b49d8db P 5d2ab23455b64a75a08d71d0db37be10: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 19:59:43.350301 25728 raft_consensus.cc:482] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:59:43.350745 25728 raft_consensus.cc:504] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.352313 25728 leader_election.cc:288] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d2ab23455b64a75a08d71d0db37be10 (127.24.69.130:33157), 016da4070e314b2786e8614d8a954ecd (127.24.69.131:42353)
I20240501 19:59:43.353021 25600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a528234b11234a6a8516d9365b49d8db" candidate_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d2ab23455b64a75a08d71d0db37be10" is_pre_election: true
I20240501 19:59:43.353313 25674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a528234b11234a6a8516d9365b49d8db" candidate_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016da4070e314b2786e8614d8a954ecd" is_pre_election: true
I20240501 19:59:43.353688 25600 raft_consensus.cc:2454] T a528234b11234a6a8516d9365b49d8db P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f6cabdca8b242d397b00a90d55ef9ec in term 0.
I20240501 19:59:43.353955 25674 raft_consensus.cc:2454] T a528234b11234a6a8516d9365b49d8db P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f6cabdca8b242d397b00a90d55ef9ec in term 0.
I20240501 19:59:43.354660 25490 leader_election.cc:302] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d2ab23455b64a75a08d71d0db37be10, 7f6cabdca8b242d397b00a90d55ef9ec; no voters:
I20240501 19:59:43.355417 25728 raft_consensus.cc:2781] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:59:43.355679 25728 raft_consensus.cc:482] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:59:43.355929 25728 raft_consensus.cc:3037] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:43.360407 25728 raft_consensus.cc:504] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.361904 25728 leader_election.cc:288] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec [CANDIDATE]: Term 1 election: Requested vote from peers 5d2ab23455b64a75a08d71d0db37be10 (127.24.69.130:33157), 016da4070e314b2786e8614d8a954ecd (127.24.69.131:42353)
I20240501 19:59:43.362520 25600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a528234b11234a6a8516d9365b49d8db" candidate_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d2ab23455b64a75a08d71d0db37be10"
I20240501 19:59:43.362960 25600 raft_consensus.cc:3037] T a528234b11234a6a8516d9365b49d8db P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:43.363040 25674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a528234b11234a6a8516d9365b49d8db" candidate_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016da4070e314b2786e8614d8a954ecd"
I20240501 19:59:43.363647 25674 raft_consensus.cc:3037] T a528234b11234a6a8516d9365b49d8db P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:43.367702 25674 raft_consensus.cc:2454] T a528234b11234a6a8516d9365b49d8db P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f6cabdca8b242d397b00a90d55ef9ec in term 1.
I20240501 19:59:43.367727 25600 raft_consensus.cc:2454] T a528234b11234a6a8516d9365b49d8db P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f6cabdca8b242d397b00a90d55ef9ec in term 1.
I20240501 19:59:43.368729 25489 leader_election.cc:302] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 016da4070e314b2786e8614d8a954ecd, 7f6cabdca8b242d397b00a90d55ef9ec; no voters:
I20240501 19:59:43.369354 25728 raft_consensus.cc:2781] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:59:43.369814 25728 raft_consensus.cc:686] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 LEADER]: Becoming Leader. State: Replica: 7f6cabdca8b242d397b00a90d55ef9ec, State: Running, Role: LEADER
I20240501 19:59:43.370622 25524 raft_consensus.cc:3037] T 47f6e1c34803471289c79387682db511 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:43.370548 25728 consensus_queue.cc:237] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.377328 25736 consensus_queue.cc:1035] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:59:43.378546 25422 catalog_manager.cc:5484] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129). New cstate: current_term: 1 leader_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } health_report { overall_health: UNKNOWN } } }
I20240501 19:59:43.389840 25736 consensus_queue.cc:1035] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:59:43.392158 25674 raft_consensus.cc:3037] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:43.399945 25726 consensus_queue.cc:1035] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:59:43.409651 25734 consensus_queue.cc:1035] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240501 19:59:43.431113 25726 raft_consensus.cc:482] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:59:43.431486 25726 raft_consensus.cc:504] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.432991 25726 leader_election.cc:288] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129:39987), 016da4070e314b2786e8614d8a954ecd (127.24.69.131:42353)
I20240501 19:59:43.433598 25524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e659b732ef82488eac87492a463bd702" candidate_uuid: "5d2ab23455b64a75a08d71d0db37be10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" is_pre_election: true
I20240501 19:59:43.433998 25673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e659b732ef82488eac87492a463bd702" candidate_uuid: "5d2ab23455b64a75a08d71d0db37be10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016da4070e314b2786e8614d8a954ecd" is_pre_election: true
I20240501 19:59:43.434120 25524 raft_consensus.cc:2454] T e659b732ef82488eac87492a463bd702 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d2ab23455b64a75a08d71d0db37be10 in term 0.
I20240501 19:59:43.434516 25673 raft_consensus.cc:2454] T e659b732ef82488eac87492a463bd702 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d2ab23455b64a75a08d71d0db37be10 in term 0.
I20240501 19:59:43.434963 25563 leader_election.cc:302] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d2ab23455b64a75a08d71d0db37be10, 7f6cabdca8b242d397b00a90d55ef9ec; no voters:
I20240501 19:59:43.435659 25726 raft_consensus.cc:2781] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:59:43.435963 25726 raft_consensus.cc:482] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:59:43.436172 25726 raft_consensus.cc:3037] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:43.441721 25726 raft_consensus.cc:504] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.442922 25726 leader_election.cc:288] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 election: Requested vote from peers 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129:39987), 016da4070e314b2786e8614d8a954ecd (127.24.69.131:42353)
I20240501 19:59:43.443603 25524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e659b732ef82488eac87492a463bd702" candidate_uuid: "5d2ab23455b64a75a08d71d0db37be10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6cabdca8b242d397b00a90d55ef9ec"
I20240501 19:59:43.443727 25673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e659b732ef82488eac87492a463bd702" candidate_uuid: "5d2ab23455b64a75a08d71d0db37be10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016da4070e314b2786e8614d8a954ecd"
I20240501 19:59:43.444043 25524 raft_consensus.cc:3037] T e659b732ef82488eac87492a463bd702 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:43.444178 25673 raft_consensus.cc:3037] T e659b732ef82488eac87492a463bd702 P 016da4070e314b2786e8614d8a954ecd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:43.448323 25524 raft_consensus.cc:2454] T e659b732ef82488eac87492a463bd702 P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d2ab23455b64a75a08d71d0db37be10 in term 1.
I20240501 19:59:43.449205 25673 raft_consensus.cc:2454] T e659b732ef82488eac87492a463bd702 P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d2ab23455b64a75a08d71d0db37be10 in term 1.
I20240501 19:59:43.449219 25563 leader_election.cc:302] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d2ab23455b64a75a08d71d0db37be10, 7f6cabdca8b242d397b00a90d55ef9ec; no voters:
I20240501 19:59:43.450089 25726 raft_consensus.cc:2781] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:59:43.450534 25726 raft_consensus.cc:686] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 [term 1 LEADER]: Becoming Leader. State: Replica: 5d2ab23455b64a75a08d71d0db37be10, State: Running, Role: LEADER
I20240501 19:59:43.450785 25736 consensus_queue.cc:1035] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:59:43.451293 25726 consensus_queue.cc:237] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.461915 25734 consensus_queue.cc:1035] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:59:43.460806 25422 catalog_manager.cc:5484] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d2ab23455b64a75a08d71d0db37be10 (127.24.69.130). New cstate: current_term: 1 leader_uuid: "5d2ab23455b64a75a08d71d0db37be10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } health_report { overall_health: UNKNOWN } } }
I20240501 19:59:43.494832 25734 raft_consensus.cc:482] T 7320413158f34e7d80d98dd88b562af7 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:59:43.495265 25734 raft_consensus.cc:504] T 7320413158f34e7d80d98dd88b562af7 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.496789 25734 leader_election.cc:288] T 7320413158f34e7d80d98dd88b562af7 P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129:39987), 016da4070e314b2786e8614d8a954ecd (127.24.69.131:42353)
I20240501 19:59:43.497929 25673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7320413158f34e7d80d98dd88b562af7" candidate_uuid: "5d2ab23455b64a75a08d71d0db37be10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016da4070e314b2786e8614d8a954ecd" is_pre_election: true
I20240501 19:59:43.498023 25525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7320413158f34e7d80d98dd88b562af7" candidate_uuid: "5d2ab23455b64a75a08d71d0db37be10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" is_pre_election: true
I20240501 19:59:43.498699 25525 raft_consensus.cc:2379] T 7320413158f34e7d80d98dd88b562af7 P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d2ab23455b64a75a08d71d0db37be10 in current term 1: Already voted for candidate 016da4070e314b2786e8614d8a954ecd in this term.
I20240501 19:59:43.499481 25563 leader_election.cc:302] T 7320413158f34e7d80d98dd88b562af7 P 5d2ab23455b64a75a08d71d0db37be10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d2ab23455b64a75a08d71d0db37be10; no voters: 016da4070e314b2786e8614d8a954ecd, 7f6cabdca8b242d397b00a90d55ef9ec
I20240501 19:59:43.500068 25734 raft_consensus.cc:3037] T 7320413158f34e7d80d98dd88b562af7 P 5d2ab23455b64a75a08d71d0db37be10 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:43.504212 25734 raft_consensus.cc:2726] T 7320413158f34e7d80d98dd88b562af7 P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:59:43.513116 25728 raft_consensus.cc:482] T 38c3d8dabd6648ff9a4262c784b88c80 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:59:43.513505 25728 raft_consensus.cc:504] T 38c3d8dabd6648ff9a4262c784b88c80 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:43.515348 25728 leader_election.cc:288] T 38c3d8dabd6648ff9a4262c784b88c80 P 7f6cabdca8b242d397b00a90d55ef9ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d2ab23455b64a75a08d71d0db37be10 (127.24.69.130:33157), 016da4070e314b2786e8614d8a954ecd (127.24.69.131:42353)
I20240501 19:59:43.516124 25600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c3d8dabd6648ff9a4262c784b88c80" candidate_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d2ab23455b64a75a08d71d0db37be10" is_pre_election: true
I20240501 19:59:43.516307 25673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c3d8dabd6648ff9a4262c784b88c80" candidate_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016da4070e314b2786e8614d8a954ecd" is_pre_election: true
I20240501 19:59:43.516770 25600 raft_consensus.cc:2379] T 38c3d8dabd6648ff9a4262c784b88c80 P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7f6cabdca8b242d397b00a90d55ef9ec in current term 1: Already voted for candidate 016da4070e314b2786e8614d8a954ecd in this term.
I20240501 19:59:43.517736 25490 leader_election.cc:302] T 38c3d8dabd6648ff9a4262c784b88c80 P 7f6cabdca8b242d397b00a90d55ef9ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f6cabdca8b242d397b00a90d55ef9ec; no voters: 016da4070e314b2786e8614d8a954ecd, 5d2ab23455b64a75a08d71d0db37be10
I20240501 19:59:43.518445 25728 raft_consensus.cc:3037] T 38c3d8dabd6648ff9a4262c784b88c80 P 7f6cabdca8b242d397b00a90d55ef9ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:43.522446 25728 raft_consensus.cc:2726] T 38c3d8dabd6648ff9a4262c784b88c80 P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:59:43.559965 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:59:43.566887 25778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:43.567765 25779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:43.568589 24854 server_base.cc:1034] running on GCE node
W20240501 19:59:43.569147 25781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:43.569939 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:59:43.570130 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:59:43.570263 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593583570252 us; error 0 us; skew 500 ppm
I20240501 19:59:43.570748 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 19:59:43.576596 24854 webserver.cc:460] Webserver started at http://127.24.69.132:37117/ using document root <none> and password file <none>
I20240501 19:59:43.577023 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 19:59:43.577183 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:59:43.577406 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:59:43.578449 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593576104309-24854-0/minicluster-data/ts-3-root/instance:
uuid: "1eff98728011449793443771c3c115be"
format_stamp: "Formatted at 2024-05-01 19:59:43 on dist-test-slave-z8x3"
I20240501 19:59:43.582930 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240501 19:59:43.586652 25786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:43.587383 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:59:43.587641 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593576104309-24854-0/minicluster-data/ts-3-root
uuid: "1eff98728011449793443771c3c115be"
format_stamp: "Formatted at 2024-05-01 19:59:43 on dist-test-slave-z8x3"
I20240501 19:59:43.587920 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593576104309-24854-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593576104309-24854-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593576104309-24854-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:59:43.607506 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:59:43.608554 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:59:43.609735 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:59:43.611871 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:59:43.612068 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:43.612315 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:59:43.612500 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:43.654544 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.132:41557
I20240501 19:59:43.654628 25848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.132:41557 every 8 connection(s)
I20240501 19:59:43.667902 25849 heartbeater.cc:347] Connected to a master server at 127.24.69.190:37003
I20240501 19:59:43.668226 25849 heartbeater.cc:464] Registering TS with master...
I20240501 19:59:43.669142 25849 heartbeater.cc:511] Master 127.24.69.190:37003 requested a full tablet report, sending...
I20240501 19:59:43.671030 25421 ts_manager.cc:194] Registered new tserver with Master: 1eff98728011449793443771c3c115be (127.24.69.132:41557)
I20240501 19:59:43.672370 25421 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55728
I20240501 19:59:43.692565 25758 consensus_queue.cc:1035] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 19:59:43.694833 25758 consensus_queue.cc:1035] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:59:43.795915 25759 consensus_queue.cc:1035] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:59:43.803243 25759 consensus_queue.cc:1035] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:59:43.819031 25857 consensus_queue.cc:1035] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:59:43.821187 25857 consensus_queue.cc:1035] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:59:43.839789 25734 consensus_queue.cc:1035] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:59:43.851869 25726 consensus_queue.cc:1035] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:59:43.894310 25758 consensus_queue.cc:1035] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:59:43.904925 25728 consensus_queue.cc:1035] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 19:59:44.674422 25849 heartbeater.cc:503] Master 127.24.69.190:37003 was elected leader, sending a full tablet report...
I20240501 19:59:45.182055 25476 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:59:45.197891 25600 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1be0bc54d8f942aa8eb1a1cdfc485a7d"
dest_uuid: "5d2ab23455b64a75a08d71d0db37be10"
mode: GRACEFUL
new_leader_uuid: "016da4070e314b2786e8614d8a954ecd"
from {username='slave'} at 127.0.0.1:45844
I20240501 19:59:45.198475 25600 raft_consensus.cc:595] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [term 1 LEADER]: Received request to transfer leadership to TS 016da4070e314b2786e8614d8a954ecd
I20240501 19:59:45.199563 25476 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 19:59:45.200054 25476 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:59:45.333257 25870 raft_consensus.cc:979] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10: : Instructing follower 016da4070e314b2786e8614d8a954ecd to start an election
I20240501 19:59:45.333685 25870 raft_consensus.cc:1067] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [term 1 LEADER]: Signalling peer 016da4070e314b2786e8614d8a954ecd to start an election
I20240501 19:59:45.335227 25673 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1be0bc54d8f942aa8eb1a1cdfc485a7d"
dest_uuid: "016da4070e314b2786e8614d8a954ecd"
from {username='slave'} at 127.0.0.1:36168
I20240501 19:59:45.335711 25673 raft_consensus.cc:482] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:59:45.335995 25673 raft_consensus.cc:3037] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:59:45.340118 25673 raft_consensus.cc:504] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } }
I20240501 19:59:45.341799 25673 leader_election.cc:288] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd [CANDIDATE]: Term 2 election: Requested vote from peers 5d2ab23455b64a75a08d71d0db37be10 (127.24.69.130:33157), 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129:39987)
I20240501 19:59:45.342564 25600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be0bc54d8f942aa8eb1a1cdfc485a7d" candidate_uuid: "016da4070e314b2786e8614d8a954ecd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5d2ab23455b64a75a08d71d0db37be10"
I20240501 19:59:45.342792 25524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be0bc54d8f942aa8eb1a1cdfc485a7d" candidate_uuid: "016da4070e314b2786e8614d8a954ecd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7f6cabdca8b242d397b00a90d55ef9ec"
I20240501 19:59:45.343202 25600 raft_consensus.cc:3032] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:59:45.343312 25524 raft_consensus.cc:3037] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:59:45.343577 25600 raft_consensus.cc:729] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5d2ab23455b64a75a08d71d0db37be10, State: Running, Role: LEADER
I20240501 19:59:45.344432 25600 consensus_queue.cc:260] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } }
I20240501 19:59:45.345688 25600 raft_consensus.cc:3037] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:59:45.349624 25524 raft_consensus.cc:2454] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 7f6cabdca8b242d397b00a90d55ef9ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 016da4070e314b2786e8614d8a954ecd in term 2.
I20240501 19:59:45.350777 25637 leader_election.cc:302] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 016da4070e314b2786e8614d8a954ecd, 7f6cabdca8b242d397b00a90d55ef9ec; no voters:
I20240501 19:59:45.351747 25600 raft_consensus.cc:2454] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 016da4070e314b2786e8614d8a954ecd in term 2.
I20240501 19:59:45.351925 25869 raft_consensus.cc:2781] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:59:45.353235 25869 raft_consensus.cc:686] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd [term 2 LEADER]: Becoming Leader. State: Replica: 016da4070e314b2786e8614d8a954ecd, State: Running, Role: LEADER
I20240501 19:59:45.354110 25869 consensus_queue.cc:237] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } }
I20240501 19:59:45.360476 25422 catalog_manager.cc:5484] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd reported cstate change: term changed from 1 to 2, leader changed from 5d2ab23455b64a75a08d71d0db37be10 (127.24.69.130) to 016da4070e314b2786e8614d8a954ecd (127.24.69.131). New cstate: current_term: 2 leader_uuid: "016da4070e314b2786e8614d8a954ecd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } health_report { overall_health: UNKNOWN } } }
I20240501 19:59:45.738389 25524 raft_consensus.cc:1261] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 7f6cabdca8b242d397b00a90d55ef9ec [term 2 FOLLOWER]: Refusing update from remote peer 016da4070e314b2786e8614d8a954ecd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 19:59:45.739470 25869 consensus_queue.cc:1035] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:59:45.749634 25599 raft_consensus.cc:1261] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [term 2 FOLLOWER]: Refusing update from remote peer 016da4070e314b2786e8614d8a954ecd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 19:59:45.751351 25874 consensus_queue.cc:1035] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:59:47.201007 25476 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:59:47.206400 25476 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:59:47.206804 25476 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:59:49.207666 25476 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:59:49.214779 25524 consensus_queue.cc:237] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } attrs { replace: true } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } } peers { permanent_uuid: "1eff98728011449793443771c3c115be" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 41557 } attrs { promote: true } }
I20240501 19:59:50.290365 25895 raft_consensus.cc:482] T 7320413158f34e7d80d98dd88b562af7 P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 016da4070e314b2786e8614d8a954ecd)
I20240501 19:59:50.291261 25895 raft_consensus.cc:504] T 7320413158f34e7d80d98dd88b562af7 P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } }
I20240501 19:59:50.292840 25599 raft_consensus.cc:1261] T a528234b11234a6a8516d9365b49d8db P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Refusing update from remote peer 7f6cabdca8b242d397b00a90d55ef9ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:59:50.293406 25895 leader_election.cc:288] T 7320413158f34e7d80d98dd88b562af7 P 7f6cabdca8b242d397b00a90d55ef9ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5d2ab23455b64a75a08d71d0db37be10 (127.24.69.130:33157), 016da4070e314b2786e8614d8a954ecd (127.24.69.131:42353)
I20240501 19:59:50.296617 25599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7320413158f34e7d80d98dd88b562af7" candidate_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5d2ab23455b64a75a08d71d0db37be10" is_pre_election: true
I20240501 19:59:50.297003 25895 consensus_queue.cc:1035] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240501 19:59:50.304337 25491 consensus_peers.cc:485] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec -> Peer 1eff98728011449793443771c3c115be (127.24.69.132:41557): Couldn't send request to peer 1eff98728011449793443771c3c115be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a528234b11234a6a8516d9365b49d8db. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:59:50.311666 25670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7320413158f34e7d80d98dd88b562af7" candidate_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "016da4070e314b2786e8614d8a954ecd" is_pre_election: true
I20240501 19:59:50.311123 25757 raft_consensus.cc:2932] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1eff98728011449793443771c3c115be (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } attrs { replace: true } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } } peers { permanent_uuid: "1eff98728011449793443771c3c115be" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 41557 } attrs { promote: true } } }
I20240501 19:59:50.311123 25671 raft_consensus.cc:1261] T a528234b11234a6a8516d9365b49d8db P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Refusing update from remote peer 7f6cabdca8b242d397b00a90d55ef9ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:59:50.320619 25489 leader_election.cc:302] T 7320413158f34e7d80d98dd88b562af7 P 7f6cabdca8b242d397b00a90d55ef9ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f6cabdca8b242d397b00a90d55ef9ec; no voters: 016da4070e314b2786e8614d8a954ecd, 5d2ab23455b64a75a08d71d0db37be10
I20240501 19:59:50.321933 25896 raft_consensus.cc:2726] T 7320413158f34e7d80d98dd88b562af7 P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:59:50.327347 25599 raft_consensus.cc:2932] T a528234b11234a6a8516d9365b49d8db P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1eff98728011449793443771c3c115be (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } attrs { replace: true } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } } peers { permanent_uuid: "1eff98728011449793443771c3c115be" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 41557 } attrs { promote: true } } }
I20240501 19:59:50.333395 25896 consensus_queue.cc:1035] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:59:50.335757 25757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:58924 (request call id 0) took 1127 ms. Trace:
I20240501 19:59:50.336449 25757 rpcz_store.cc:276] 0501 19:59:49.208584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:59:49.208737 (+ 153us) service_pool.cc:225] Handling call
0501 19:59:50.241926 (+1033189us) log.cc:842] Serialized 506 byte log entry
0501 19:59:50.335651 (+ 93725us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":190,"raft.queue_time_us":5497,"raft.run_cpu_time_us":1598,"raft.run_wall_time_us":3553,"thread_start_us":9321,"threads_started":3,"wal-append.queue_time_us":6870}
I20240501 19:59:50.344933 25476 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:59:50.345551 25476 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:59:50.348506 25674 raft_consensus.cc:2932] T a528234b11234a6a8516d9365b49d8db P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1eff98728011449793443771c3c115be (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } attrs { replace: true } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } } peers { permanent_uuid: "1eff98728011449793443771c3c115be" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 41557 } attrs { promote: true } } }
I20240501 19:59:50.360324 25422 catalog_manager.cc:5484] T a528234b11234a6a8516d9365b49d8db P 5d2ab23455b64a75a08d71d0db37be10 reported cstate change: config changed from index -1 to 2, NON_VOTER 1eff98728011449793443771c3c115be (127.24.69.132) added. New cstate: current_term: 1 leader_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } attrs { replace: true } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } } peers { permanent_uuid: "1eff98728011449793443771c3c115be" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 41557 } attrs { promote: true } } }
I20240501 19:59:50.371675 25599 consensus_queue.cc:237] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } attrs { replace: true } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } } peers { permanent_uuid: "1eff98728011449793443771c3c115be" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 41557 } attrs { promote: true } }
I20240501 19:59:50.388979 25674 raft_consensus.cc:1261] T 47f6e1c34803471289c79387682db511 P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Refusing update from remote peer 5d2ab23455b64a75a08d71d0db37be10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:59:50.389461 25523 raft_consensus.cc:1261] T 47f6e1c34803471289c79387682db511 P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Refusing update from remote peer 5d2ab23455b64a75a08d71d0db37be10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:59:50.390319 25893 consensus_queue.cc:1035] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 19:59:50.391391 25899 consensus_queue.cc:1035] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 19:59:50.395452 25566 consensus_peers.cc:485] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 -> Peer 1eff98728011449793443771c3c115be (127.24.69.132:41557): Couldn't send request to peer 1eff98728011449793443771c3c115be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 47f6e1c34803471289c79387682db511. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:59:50.400688 25900 raft_consensus.cc:2932] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1eff98728011449793443771c3c115be (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } attrs { replace: true } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } } peers { permanent_uuid: "1eff98728011449793443771c3c115be" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 41557 } attrs { promote: true } } }
I20240501 19:59:50.403251 25674 raft_consensus.cc:2932] T 47f6e1c34803471289c79387682db511 P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1eff98728011449793443771c3c115be (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } attrs { replace: true } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } } peers { permanent_uuid: "1eff98728011449793443771c3c115be" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 41557 } attrs { promote: true } } }
I20240501 19:59:50.408955 25523 raft_consensus.cc:2932] T 47f6e1c34803471289c79387682db511 P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1eff98728011449793443771c3c115be (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } attrs { replace: true } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } } peers { permanent_uuid: "1eff98728011449793443771c3c115be" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 41557 } attrs { promote: true } } }
I20240501 19:59:50.413587 25421 catalog_manager.cc:5484] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 reported cstate change: config changed from index -1 to 2, NON_VOTER 1eff98728011449793443771c3c115be (127.24.69.132) added. New cstate: current_term: 1 leader_uuid: "5d2ab23455b64a75a08d71d0db37be10" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eff98728011449793443771c3c115be" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 41557 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:59:50.415256 25599 consensus_queue.cc:237] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } attrs { replace: true } } peers { permanent_uuid: "1eff98728011449793443771c3c115be" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 41557 } attrs { promote: true } }
W20240501 19:59:50.423507 25566 consensus_peers.cc:485] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 -> Peer 1eff98728011449793443771c3c115be (127.24.69.132:41557): Couldn't send request to peer 1eff98728011449793443771c3c115be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e659b732ef82488eac87492a463bd702. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:59:50.424453 25674 raft_consensus.cc:1261] T e659b732ef82488eac87492a463bd702 P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Refusing update from remote peer 5d2ab23455b64a75a08d71d0db37be10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:59:50.425071 25523 raft_consensus.cc:1261] T e659b732ef82488eac87492a463bd702 P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Refusing update from remote peer 5d2ab23455b64a75a08d71d0db37be10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:59:50.425704 25899 consensus_queue.cc:1035] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:59:50.427480 25900 consensus_queue.cc:1035] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:59:50.434855 25899 raft_consensus.cc:2932] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1eff98728011449793443771c3c115be (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } attrs { replace: true } } peers { permanent_uuid: "1eff98728011449793443771c3c115be" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 41557 } attrs { promote: true } } }
I20240501 19:59:50.436748 25523 raft_consensus.cc:2932] T e659b732ef82488eac87492a463bd702 P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1eff98728011449793443771c3c115be (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } attrs { replace: true } } peers { permanent_uuid: "1eff98728011449793443771c3c115be" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 41557 } attrs { promote: true } } }
I20240501 19:59:50.437546 25673 raft_consensus.cc:2932] T e659b732ef82488eac87492a463bd702 P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1eff98728011449793443771c3c115be (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } attrs { replace: true } } peers { permanent_uuid: "1eff98728011449793443771c3c115be" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 41557 } attrs { promote: true } } }
I20240501 19:59:50.447463 25421 catalog_manager.cc:5484] T e659b732ef82488eac87492a463bd702 P 7f6cabdca8b242d397b00a90d55ef9ec reported cstate change: config changed from index -1 to 2, NON_VOTER 1eff98728011449793443771c3c115be (127.24.69.132) added. New cstate: current_term: 1 leader_uuid: "5d2ab23455b64a75a08d71d0db37be10" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } attrs { replace: true } } peers { permanent_uuid: "1eff98728011449793443771c3c115be" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 41557 } attrs { promote: true } } }
I20240501 19:59:50.473177 25673 consensus_queue.cc:237] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } attrs { replace: true } } peers { permanent_uuid: "1eff98728011449793443771c3c115be" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 41557 } attrs { promote: true } }
I20240501 19:59:50.480196 25523 raft_consensus.cc:1261] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 7f6cabdca8b242d397b00a90d55ef9ec [term 2 FOLLOWER]: Refusing update from remote peer 016da4070e314b2786e8614d8a954ecd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:59:50.480978 25599 raft_consensus.cc:1261] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [term 2 FOLLOWER]: Refusing update from remote peer 016da4070e314b2786e8614d8a954ecd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:59:50.482822 25903 consensus_queue.cc:1035] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 19:59:50.483860 25920 consensus_queue.cc:1035] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20240501 19:59:50.490792 25640 consensus_peers.cc:485] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd -> Peer 1eff98728011449793443771c3c115be (127.24.69.132:41557): Couldn't send request to peer 1eff98728011449793443771c3c115be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1be0bc54d8f942aa8eb1a1cdfc485a7d. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:59:50.491690 25920 raft_consensus.cc:2932] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1eff98728011449793443771c3c115be (127.24.69.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } attrs { replace: true } } peers { permanent_uuid: "1eff98728011449793443771c3c115be" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 41557 } attrs { promote: true } } }
I20240501 19:59:50.493489 25599 raft_consensus.cc:2932] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1eff98728011449793443771c3c115be (127.24.69.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } attrs { replace: true } } peers { permanent_uuid: "1eff98728011449793443771c3c115be" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 41557 } attrs { promote: true } } }
I20240501 19:59:50.495492 25523 raft_consensus.cc:2932] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 7f6cabdca8b242d397b00a90d55ef9ec [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1eff98728011449793443771c3c115be (127.24.69.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } } peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } attrs { replace: true } } peers { permanent_uuid: "1eff98728011449793443771c3c115be" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 41557 } attrs { promote: true } } }
I20240501 19:59:50.506057 25421 catalog_manager.cc:5484] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd reported cstate change: config changed from index -1 to 3, NON_VOTER 1eff98728011449793443771c3c115be (127.24.69.132) added. New cstate: current_term: 2 leader_uuid: "016da4070e314b2786e8614d8a954ecd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "016da4070e314b2786e8614d8a954ecd" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 42353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d2ab23455b64a75a08d71d0db37be10" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f6cabdca8b242d397b00a90d55ef9ec" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 39987 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eff98728011449793443771c3c115be" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 41557 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:59:50.677769 24854 tablet_server.cc:178] TabletServer@127.24.69.129:0 shutting down...
W20240501 19:59:50.681223 25407 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 7 similar messages]
W20240501 19:59:50.684564 25471 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 19:59:50.684973 25471 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240501 19:59:50.710610 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:59:50.711541 24854 tablet_replica.cc:330] T 38c3d8dabd6648ff9a4262c784b88c80 P 7f6cabdca8b242d397b00a90d55ef9ec: stopping tablet replica
I20240501 19:59:50.712281 24854 raft_consensus.cc:2229] T 38c3d8dabd6648ff9a4262c784b88c80 P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:59:50.712793 24854 raft_consensus.cc:2258] T 38c3d8dabd6648ff9a4262c784b88c80 P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:50.715235 24854 tablet_replica.cc:330] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec: stopping tablet replica
I20240501 19:59:50.715759 24854 raft_consensus.cc:2229] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:59:50.716781 24854 raft_consensus.cc:2258] T a528234b11234a6a8516d9365b49d8db P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:50.718930 24854 tablet_replica.cc:330] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 7f6cabdca8b242d397b00a90d55ef9ec: stopping tablet replica
I20240501 19:59:50.719579 24854 raft_consensus.cc:2229] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 7f6cabdca8b242d397b00a90d55ef9ec [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:59:50.719985 24854 raft_consensus.cc:2258] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 7f6cabdca8b242d397b00a90d55ef9ec [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:50.722806 24854 tablet_replica.cc:330] T 7320413158f34e7d80d98dd88b562af7 P 7f6cabdca8b242d397b00a90d55ef9ec: stopping tablet replica
I20240501 19:59:50.723321 24854 raft_consensus.cc:2229] T 7320413158f34e7d80d98dd88b562af7 P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:59:50.723764 24854 raft_consensus.cc:2258] T 7320413158f34e7d80d98dd88b562af7 P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:50.725612 24854 tablet_replica.cc:330] T e659b732ef82488eac87492a463bd702 P 7f6cabdca8b242d397b00a90d55ef9ec: stopping tablet replica
I20240501 19:59:50.726227 24854 raft_consensus.cc:2229] T e659b732ef82488eac87492a463bd702 P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:59:50.726661 24854 raft_consensus.cc:2258] T e659b732ef82488eac87492a463bd702 P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:50.728986 24854 tablet_replica.cc:330] T 47f6e1c34803471289c79387682db511 P 7f6cabdca8b242d397b00a90d55ef9ec: stopping tablet replica
I20240501 19:59:50.729524 24854 raft_consensus.cc:2229] T 47f6e1c34803471289c79387682db511 P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:59:50.729899 24854 raft_consensus.cc:2258] T 47f6e1c34803471289c79387682db511 P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:50.732053 24854 tablet_replica.cc:330] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec: stopping tablet replica
I20240501 19:59:50.732695 24854 raft_consensus.cc:2229] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:59:50.733479 24854 raft_consensus.cc:2258] T ba5ada435b794cc194e3cbff0d994915 P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:50.735744 24854 tablet_replica.cc:330] T 3a18557702154a5a91ff0d36e4df951d P 7f6cabdca8b242d397b00a90d55ef9ec: stopping tablet replica
I20240501 19:59:50.736299 24854 raft_consensus.cc:2229] T 3a18557702154a5a91ff0d36e4df951d P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:59:50.736752 24854 raft_consensus.cc:2258] T 3a18557702154a5a91ff0d36e4df951d P 7f6cabdca8b242d397b00a90d55ef9ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:50.767480 24854 tablet_server.cc:194] TabletServer@127.24.69.129:0 shutdown complete.
I20240501 19:59:50.785977 24854 tablet_server.cc:178] TabletServer@127.24.69.130:0 shutting down...
W20240501 19:59:50.795382 25471 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 19:59:50.795668 25471 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 19:59:50.803126 25471 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 19:59:50.803396 25471 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 19:59:50.808900 25563 consensus_peers.cc:485] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10 -> Peer 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129:39987): Couldn't send request to peer 7f6cabdca8b242d397b00a90d55ef9ec. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:39987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:59:50.817464 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:59:50.818209 24854 tablet_replica.cc:330] T ba5ada435b794cc194e3cbff0d994915 P 5d2ab23455b64a75a08d71d0db37be10: stopping tablet replica
I20240501 19:59:50.819105 24854 raft_consensus.cc:2229] T ba5ada435b794cc194e3cbff0d994915 P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:59:50.819598 24854 raft_consensus.cc:2258] T ba5ada435b794cc194e3cbff0d994915 P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:50.821887 24854 tablet_replica.cc:330] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10: stopping tablet replica
I20240501 19:59:50.822569 24854 raft_consensus.cc:2229] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:59:50.823495 24854 raft_consensus.cc:2258] T 3a18557702154a5a91ff0d36e4df951d P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:50.825774 24854 tablet_replica.cc:330] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10: stopping tablet replica
I20240501 19:59:50.826435 24854 raft_consensus.cc:2229] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:59:50.827363 24854 raft_consensus.cc:2258] T 47f6e1c34803471289c79387682db511 P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:50.829984 24854 tablet_replica.cc:330] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10: stopping tablet replica
I20240501 19:59:50.830966 24854 raft_consensus.cc:2229] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:59:50.833259 24854 raft_consensus.cc:2258] T e659b732ef82488eac87492a463bd702 P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 19:59:50.833920 25639 consensus_peers.cc:485] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd -> Peer 5d2ab23455b64a75a08d71d0db37be10 (127.24.69.130:33157): Couldn't send request to peer 5d2ab23455b64a75a08d71d0db37be10. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240501 19:59:50.834971 25637 consensus_peers.cc:485] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd -> Peer 7f6cabdca8b242d397b00a90d55ef9ec (127.24.69.129:39987): Couldn't send request to peer 7f6cabdca8b242d397b00a90d55ef9ec. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:39987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:59:50.836539 24854 tablet_replica.cc:330] T a528234b11234a6a8516d9365b49d8db P 5d2ab23455b64a75a08d71d0db37be10: stopping tablet replica
I20240501 19:59:50.837116 24854 raft_consensus.cc:2229] T a528234b11234a6a8516d9365b49d8db P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:59:50.837528 24854 raft_consensus.cc:2258] T a528234b11234a6a8516d9365b49d8db P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:50.839741 24854 tablet_replica.cc:330] T 38c3d8dabd6648ff9a4262c784b88c80 P 5d2ab23455b64a75a08d71d0db37be10: stopping tablet replica
I20240501 19:59:50.840193 24854 raft_consensus.cc:2229] T 38c3d8dabd6648ff9a4262c784b88c80 P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:59:50.840672 24854 raft_consensus.cc:2258] T 38c3d8dabd6648ff9a4262c784b88c80 P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:50.842417 24854 tablet_replica.cc:330] T 7320413158f34e7d80d98dd88b562af7 P 5d2ab23455b64a75a08d71d0db37be10: stopping tablet replica
I20240501 19:59:50.842847 24854 raft_consensus.cc:2229] T 7320413158f34e7d80d98dd88b562af7 P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:59:50.843226 24854 raft_consensus.cc:2258] T 7320413158f34e7d80d98dd88b562af7 P 5d2ab23455b64a75a08d71d0db37be10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:50.844861 24854 tablet_replica.cc:330] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10: stopping tablet replica
I20240501 19:59:50.845256 24854 raft_consensus.cc:2229] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:59:50.845665 24854 raft_consensus.cc:2258] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 5d2ab23455b64a75a08d71d0db37be10 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240501 19:59:50.867450 25639 consensus_peers.cc:485] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd -> Peer 5d2ab23455b64a75a08d71d0db37be10 (127.24.69.130:33157): Couldn't send request to peer 5d2ab23455b64a75a08d71d0db37be10. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.130:33157: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:59:50.867880 24854 tablet_server.cc:194] TabletServer@127.24.69.130:0 shutdown complete.
I20240501 19:59:50.886396 24854 tablet_server.cc:178] TabletServer@127.24.69.131:0 shutting down...
W20240501 19:59:50.891505 25471 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 19:59:50.891842 25471 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240501 19:59:50.908761 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:59:50.909376 24854 tablet_replica.cc:330] T ba5ada435b794cc194e3cbff0d994915 P 016da4070e314b2786e8614d8a954ecd: stopping tablet replica
I20240501 19:59:50.909987 24854 raft_consensus.cc:2229] T ba5ada435b794cc194e3cbff0d994915 P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:59:50.910398 24854 raft_consensus.cc:2258] T ba5ada435b794cc194e3cbff0d994915 P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:50.912089 24854 tablet_replica.cc:330] T 47f6e1c34803471289c79387682db511 P 016da4070e314b2786e8614d8a954ecd: stopping tablet replica
I20240501 19:59:50.912665 24854 raft_consensus.cc:2229] T 47f6e1c34803471289c79387682db511 P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:59:50.913031 24854 raft_consensus.cc:2258] T 47f6e1c34803471289c79387682db511 P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:50.915539 24854 tablet_replica.cc:330] T e659b732ef82488eac87492a463bd702 P 016da4070e314b2786e8614d8a954ecd: stopping tablet replica
I20240501 19:59:50.915985 24854 raft_consensus.cc:2229] T e659b732ef82488eac87492a463bd702 P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:59:50.916416 24854 raft_consensus.cc:2258] T e659b732ef82488eac87492a463bd702 P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:50.918294 24854 tablet_replica.cc:330] T 3a18557702154a5a91ff0d36e4df951d P 016da4070e314b2786e8614d8a954ecd: stopping tablet replica
I20240501 19:59:50.918716 24854 raft_consensus.cc:2229] T 3a18557702154a5a91ff0d36e4df951d P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:59:50.919034 24854 raft_consensus.cc:2258] T 3a18557702154a5a91ff0d36e4df951d P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:50.920532 24854 tablet_replica.cc:330] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd: stopping tablet replica
I20240501 19:59:50.920941 24854 raft_consensus.cc:2229] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:59:50.921567 24854 raft_consensus.cc:2258] T 38c3d8dabd6648ff9a4262c784b88c80 P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:50.922968 24854 tablet_replica.cc:330] T a528234b11234a6a8516d9365b49d8db P 016da4070e314b2786e8614d8a954ecd: stopping tablet replica
I20240501 19:59:50.923352 24854 raft_consensus.cc:2229] T a528234b11234a6a8516d9365b49d8db P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:59:50.923693 24854 raft_consensus.cc:2258] T a528234b11234a6a8516d9365b49d8db P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:50.925488 24854 tablet_replica.cc:330] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd: stopping tablet replica
I20240501 19:59:50.925899 24854 raft_consensus.cc:2229] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd [term 2 LEADER]: Raft consensus shutting down.
I20240501 19:59:50.926621 24854 raft_consensus.cc:2258] T 1be0bc54d8f942aa8eb1a1cdfc485a7d P 016da4070e314b2786e8614d8a954ecd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:50.928332 24854 tablet_replica.cc:330] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd: stopping tablet replica
I20240501 19:59:50.928697 24854 raft_consensus.cc:2229] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:59:50.929277 24854 raft_consensus.cc:2258] T 7320413158f34e7d80d98dd88b562af7 P 016da4070e314b2786e8614d8a954ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:50.954254 24854 tablet_server.cc:194] TabletServer@127.24.69.131:0 shutdown complete.
I20240501 19:59:50.968390 24854 tablet_server.cc:178] TabletServer@127.24.69.132:0 shutting down...
I20240501 19:59:50.989076 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
W20240501 19:59:51.319180 25845 debug-util.cc:398] Leaking SignalData structure 0x7b080015d940 after lost signal to thread 24857
W20240501 19:59:51.319993 25845 debug-util.cc:398] Leaking SignalData structure 0x7b080015f020 after lost signal to thread 25457
W20240501 19:59:51.904322 25471 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.24.69.131:42353: connect: Connection refused (error 111)
W20240501 19:59:51.910499 25471 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.24.69.131:42353: connect: Connection refused (error 111)
W20240501 19:59:51.910851 25471 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.24.69.131:42353: connect: Connection refused (error 111)
W20240501 19:59:51.915309 25471 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.24.69.130:33157: connect: Connection refused (error 111)
W20240501 19:59:51.915616 25471 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.24.69.130:33157: connect: Connection refused (error 111)
W20240501 19:59:51.919785 25471 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.24.69.131:42353: connect: Connection refused (error 111)
W20240501 19:59:51.920094 25471 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.24.69.131:42353: connect: Connection refused (error 111)
W20240501 19:59:51.924149 25471 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.24.69.130:33157: connect: Connection refused (error 111)
W20240501 19:59:51.924451 25471 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.24.69.130:33157: connect: Connection refused (error 111)
W20240501 19:59:52.002441 24854 thread.cc:536] Waited for 1000ms trying to join with diag-logger (tid 25845)
I20240501 19:59:52.346508 25476 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:59:52.351855 25476 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:59:52.352247 25476 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240501 19:59:52.944082 25471 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.24.69.129:39987: connect: Connection refused (error 111)
W20240501 19:59:52.950948 25471 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.24.69.129:39987: connect: Connection refused (error 111)
W20240501 19:59:52.951267 25471 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.24.69.129:39987: connect: Connection refused (error 111)
W20240501 19:59:52.955543 25471 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.24.69.131:42353: connect: Connection refused (error 111)
W20240501 19:59:52.955853 25471 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.24.69.131:42353: connect: Connection refused (error 111)
W20240501 19:59:52.960304 25471 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.24.69.131:42353: connect: Connection refused (error 111)
W20240501 19:59:52.960613 25471 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.24.69.131:42353: connect: Connection refused (error 111)
W20240501 19:59:52.965010 25471 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.24.69.130:33157: connect: Connection refused (error 111)
W20240501 19:59:52.965315 25471 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.24.69.130:33157: connect: Connection refused (error 111)
W20240501 19:59:53.002854 24854 thread.cc:536] Waited for 2000ms trying to join with diag-logger (tid 25845)
W20240501 19:59:53.984916 25471 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.24.69.129:39987: connect: Connection refused (error 111)
W20240501 19:59:53.991950 25471 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.24.69.129:39987: connect: Connection refused (error 111)
W20240501 19:59:53.992293 25471 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.24.69.129:39987: connect: Connection refused (error 111)
W20240501 19:59:53.996433 25471 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.24.69.129:39987: connect: Connection refused (error 111)
W20240501 19:59:53.996644 25471 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.24.69.129:39987: connect: Connection refused (error 111)
W20240501 19:59:54.000664 25471 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.24.69.131:42353: connect: Connection refused (error 111)
W20240501 19:59:54.000918 25471 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.24.69.131:42353: connect: Connection refused (error 111)
W20240501 19:59:54.003302 24854 thread.cc:536] Waited for 3000ms trying to join with diag-logger (tid 25845)
W20240501 19:59:54.005398 25471 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.24.69.130:33157: connect: Connection refused (error 111)
W20240501 19:59:54.005642 25471 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.24.69.130:33157: connect: Connection refused (error 111)
I20240501 19:59:54.290709 24854 tablet_server.cc:194] TabletServer@127.24.69.132:0 shutdown complete.
I20240501 19:59:54.298071 24854 master.cc:537] Master@127.24.69.190:37003 shutting down...
I20240501 19:59:54.315402 24854 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b42b983e255d44bdb0554d040121d15f [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:59:54.315901 24854 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b42b983e255d44bdb0554d040121d15f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:54.316198 24854 tablet_replica.cc:330] T 00000000000000000000000000000000 P b42b983e255d44bdb0554d040121d15f: stopping tablet replica
I20240501 19:59:54.333251 24854 master.cc:552] Master@127.24.69.190:37003 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (13320 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240501 19:59:54.371342 24854 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.69.190:35029,127.24.69.189:45901,127.24.69.188:34347
I20240501 19:59:54.372589 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:59:54.378580 25936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:54.379369 25937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:54.380120 25939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:54.380741 24854 server_base.cc:1034] running on GCE node
I20240501 19:59:54.381608 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:59:54.381801 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:59:54.381945 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593594381927 us; error 0 us; skew 500 ppm
I20240501 19:59:54.382419 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 19:59:54.388597 24854 webserver.cc:460] Webserver started at http://127.24.69.190:32923/ using document root <none> and password file <none>
I20240501 19:59:54.389061 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 19:59:54.389228 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:59:54.389497 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:59:54.390578 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-0-root/instance:
uuid: "0e7620906a374531b01c4c1a2e56c8da"
format_stamp: "Formatted at 2024-05-01 19:59:54 on dist-test-slave-z8x3"
I20240501 19:59:54.394685 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 19:59:54.397912 25944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:54.398633 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:59:54.398890 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-0-root
uuid: "0e7620906a374531b01c4c1a2e56c8da"
format_stamp: "Formatted at 2024-05-01 19:59:54 on dist-test-slave-z8x3"
I20240501 19:59:54.399132 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:59:54.415635 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:59:54.416815 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:59:54.451541 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.190:35029
I20240501 19:59:54.451627 25995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.190:35029 every 8 connection(s)
I20240501 19:59:54.455235 25996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:54.455387 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:59:54.461282 25998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:54.461943 25996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } has no permanent_uuid. Determining permanent_uuid...
W20240501 19:59:54.462146 25999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:54.465690 24854 server_base.cc:1034] running on GCE node
W20240501 19:59:54.466312 26001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:54.467475 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:59:54.467741 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:59:54.467952 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593594467932 us; error 0 us; skew 500 ppm
I20240501 19:59:54.468631 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 19:59:54.471050 24854 webserver.cc:460] Webserver started at http://127.24.69.189:37521/ using document root <none> and password file <none>
I20240501 19:59:54.471637 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 19:59:54.471877 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:59:54.472179 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:59:54.473491 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-1-root/instance:
uuid: "08ddae5019c54872b0b497b9105de9ed"
format_stamp: "Formatted at 2024-05-01 19:59:54 on dist-test-slave-z8x3"
I20240501 19:59:54.476325 25996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:59:54.478776 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
W20240501 19:59:54.480994 25996 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.24.69.189:45901: Network error: Client connection negotiation failed: client connection to 127.24.69.189:45901: connect: Connection refused (error 111)
I20240501 19:59:54.482693 26009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:54.483404 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:59:54.483685 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-1-root
uuid: "08ddae5019c54872b0b497b9105de9ed"
format_stamp: "Formatted at 2024-05-01 19:59:54 on dist-test-slave-z8x3"
I20240501 19:59:54.483945 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:59:54.499384 25996 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } attempt: 1
W20240501 19:59:54.502887 25996 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.24.69.189:45901: Network error: Client connection negotiation failed: client connection to 127.24.69.189:45901: connect: Connection refused (error 111)
I20240501 19:59:54.524967 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:59:54.526042 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:59:54.541309 25996 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } attempt: 2
W20240501 19:59:54.545588 25996 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.24.69.189:45901: Network error: Client connection negotiation failed: client connection to 127.24.69.189:45901: connect: Connection refused (error 111)
I20240501 19:59:54.563736 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.189:45901
I20240501 19:59:54.563827 26060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.189:45901 every 8 connection(s)
I20240501 19:59:54.567302 26061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:54.567495 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:59:54.573732 26063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:54.575585 26061 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } has no permanent_uuid. Determining permanent_uuid...
W20240501 19:59:54.575846 26064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:54.576664 26066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:54.577286 24854 server_base.cc:1034] running on GCE node
I20240501 19:59:54.578222 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:59:54.578524 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:59:54.578728 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593594578711 us; error 0 us; skew 500 ppm
I20240501 19:59:54.579214 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 19:59:54.582024 24854 webserver.cc:460] Webserver started at http://127.24.69.188:39567/ using document root <none> and password file <none>
I20240501 19:59:54.582633 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 19:59:54.582859 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:59:54.583146 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:59:54.584282 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-2-root/instance:
uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186"
format_stamp: "Formatted at 2024-05-01 19:59:54 on dist-test-slave-z8x3"
I20240501 19:59:54.588186 26061 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:59:54.591092 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20240501 19:59:54.595113 26073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:54.595893 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:59:54.596159 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-2-root
uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186"
format_stamp: "Formatted at 2024-05-01 19:59:54 on dist-test-slave-z8x3"
I20240501 19:59:54.596459 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:59:54.601548 26061 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } has no permanent_uuid. Determining permanent_uuid...
W20240501 19:59:54.606109 26061 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.24.69.188:34347: Network error: Client connection negotiation failed: client connection to 127.24.69.188:34347: connect: Connection refused (error 111)
I20240501 19:59:54.622005 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:59:54.622534 26061 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } attempt: 1
I20240501 19:59:54.623255 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
W20240501 19:59:54.626502 26061 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.24.69.188:34347: Network error: Client connection negotiation failed: client connection to 127.24.69.188:34347: connect: Connection refused (error 111)
I20240501 19:59:54.636020 25996 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } attempt: 3
I20240501 19:59:54.648516 25996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } has no permanent_uuid. Determining permanent_uuid...
W20240501 19:59:54.653028 25996 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.24.69.188:34347: Network error: Client connection negotiation failed: client connection to 127.24.69.188:34347: connect: Connection refused (error 111)
I20240501 19:59:54.667445 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.188:34347
I20240501 19:59:54.667546 26125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.188:34347 every 8 connection(s)
I20240501 19:59:54.670396 24854 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 19:59:54.670957 26126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:54.676417 26126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:59:54.685868 26126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:59:54.693943 26126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:59:54.697434 25996 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } attempt: 1
I20240501 19:59:54.700971 26061 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } attempt: 2
I20240501 19:59:54.713307 26126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186: Bootstrap starting.
I20240501 19:59:54.715078 25996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da: Bootstrap starting.
I20240501 19:59:54.717850 26126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:54.720083 26061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed: Bootstrap starting.
I20240501 19:59:54.720888 25996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:54.722101 26126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186: No bootstrap required, opened a new log
I20240501 19:59:54.724228 26126 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7620906a374531b01c4c1a2e56c8da" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } } peers { permanent_uuid: "08ddae5019c54872b0b497b9105de9ed" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } } peers { permanent_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } }
I20240501 19:59:54.724929 26126 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:54.725167 26126 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0844a5f3d7c4428bc1c0c10ba3a4186, State: Initialized, Role: FOLLOWER
I20240501 19:59:54.725909 26061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:54.726040 25996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da: No bootstrap required, opened a new log
I20240501 19:59:54.725757 26126 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7620906a374531b01c4c1a2e56c8da" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } } peers { permanent_uuid: "08ddae5019c54872b0b497b9105de9ed" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } } peers { permanent_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } }
I20240501 19:59:54.727802 26134 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7620906a374531b01c4c1a2e56c8da" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } } peers { permanent_uuid: "08ddae5019c54872b0b497b9105de9ed" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } } peers { permanent_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } } }
I20240501 19:59:54.728694 26134 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:59:54.728741 25996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7620906a374531b01c4c1a2e56c8da" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } } peers { permanent_uuid: "08ddae5019c54872b0b497b9105de9ed" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } } peers { permanent_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } }
I20240501 19:59:54.729450 26126 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:59:54.729784 25996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:54.730087 25996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e7620906a374531b01c4c1a2e56c8da, State: Initialized, Role: FOLLOWER
I20240501 19:59:54.730733 25996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7620906a374531b01c4c1a2e56c8da" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } } peers { permanent_uuid: "08ddae5019c54872b0b497b9105de9ed" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } } peers { permanent_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } }
I20240501 19:59:54.731534 26061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed: No bootstrap required, opened a new log
I20240501 19:59:54.734793 26061 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7620906a374531b01c4c1a2e56c8da" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } } peers { permanent_uuid: "08ddae5019c54872b0b497b9105de9ed" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } } peers { permanent_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } }
I20240501 19:59:54.735792 26061 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:54.736220 26061 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ddae5019c54872b0b497b9105de9ed, State: Initialized, Role: FOLLOWER
I20240501 19:59:54.737085 26137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7620906a374531b01c4c1a2e56c8da" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } } peers { permanent_uuid: "08ddae5019c54872b0b497b9105de9ed" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } } peers { permanent_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } } }
I20240501 19:59:54.737164 26061 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7620906a374531b01c4c1a2e56c8da" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } } peers { permanent_uuid: "08ddae5019c54872b0b497b9105de9ed" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } } peers { permanent_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } }
I20240501 19:59:54.737970 26137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:59:54.738636 25996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:59:54.741087 26143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7620906a374531b01c4c1a2e56c8da" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } } peers { permanent_uuid: "08ddae5019c54872b0b497b9105de9ed" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } } peers { permanent_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } } }
I20240501 19:59:54.742166 26143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:59:54.742467 26061 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [sys.catalog]: configured and running, proceeding with master startup.
W20240501 19:59:54.753297 26153 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:59:54.753563 26153 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 19:59:54.758589 24854 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20240501 19:59:54.761678 26164 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:59:54.761960 26164 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 19:59:54.762696 24854 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240501 19:59:54.764010 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:59:54.765456 26170 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:59:54.765715 26170 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240501 19:59:54.770453 26171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:54.771507 26172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:54.772670 26174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:54.773102 24854 server_base.cc:1034] running on GCE node
I20240501 19:59:54.773772 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:59:54.773950 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:59:54.774076 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593594774066 us; error 0 us; skew 500 ppm
I20240501 19:59:54.774546 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 19:59:54.776800 24854 webserver.cc:460] Webserver started at http://127.24.69.129:40125/ using document root <none> and password file <none>
I20240501 19:59:54.777192 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 19:59:54.777355 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:59:54.777561 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:59:54.778589 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-0-root/instance:
uuid: "88ecb0a11a8a4950a8550193a3f81d9f"
format_stamp: "Formatted at 2024-05-01 19:59:54 on dist-test-slave-z8x3"
I20240501 19:59:54.782737 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 19:59:54.785710 26179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:54.786509 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:59:54.786767 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-0-root
uuid: "88ecb0a11a8a4950a8550193a3f81d9f"
format_stamp: "Formatted at 2024-05-01 19:59:54 on dist-test-slave-z8x3"
I20240501 19:59:54.787010 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:59:54.801146 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:59:54.802212 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:59:54.803494 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:59:54.805681 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:59:54.805871 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:54.806085 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:59:54.806257 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:54.844178 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.129:46143
I20240501 19:59:54.844427 26241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.129:46143 every 8 connection(s)
I20240501 19:59:54.853928 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 19:59:54.867177 26243 heartbeater.cc:347] Connected to a master server at 127.24.69.188:34347
I20240501 19:59:54.867607 26243 heartbeater.cc:464] Registering TS with master...
I20240501 19:59:54.868554 26243 heartbeater.cc:511] Master 127.24.69.188:34347 requested a full tablet report, sending...
I20240501 19:59:54.869241 26242 heartbeater.cc:347] Connected to a master server at 127.24.69.189:45901
I20240501 19:59:54.869697 26242 heartbeater.cc:464] Registering TS with master...
I20240501 19:59:54.870736 26242 heartbeater.cc:511] Master 127.24.69.189:45901 requested a full tablet report, sending...
I20240501 19:59:54.871614 26091 ts_manager.cc:194] Registered new tserver with Master: 88ecb0a11a8a4950a8550193a3f81d9f (127.24.69.129:46143)
I20240501 19:59:54.874572 26026 ts_manager.cc:194] Registered new tserver with Master: 88ecb0a11a8a4950a8550193a3f81d9f (127.24.69.129:46143)
W20240501 19:59:54.876770 26250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:54.878008 26244 heartbeater.cc:347] Connected to a master server at 127.24.69.190:35029
I20240501 19:59:54.878391 26244 heartbeater.cc:464] Registering TS with master...
W20240501 19:59:54.878423 26249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:54.879460 26244 heartbeater.cc:511] Master 127.24.69.190:35029 requested a full tablet report, sending...
W20240501 19:59:54.881062 26252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:54.881632 24854 server_base.cc:1034] running on GCE node
I20240501 19:59:54.881880 25961 ts_manager.cc:194] Registered new tserver with Master: 88ecb0a11a8a4950a8550193a3f81d9f (127.24.69.129:46143)
I20240501 19:59:54.882453 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:59:54.882735 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:59:54.882926 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593594882901 us; error 0 us; skew 500 ppm
I20240501 19:59:54.883464 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 19:59:54.885766 24854 webserver.cc:460] Webserver started at http://127.24.69.130:39963/ using document root <none> and password file <none>
I20240501 19:59:54.886202 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 19:59:54.886386 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:59:54.886623 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:59:54.887779 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-1-root/instance:
uuid: "58c44d190aef48c28fc8e3269ee2952a"
format_stamp: "Formatted at 2024-05-01 19:59:54 on dist-test-slave-z8x3"
I20240501 19:59:54.891947 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240501 19:59:54.894786 26257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:54.895473 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:59:54.895733 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-1-root
uuid: "58c44d190aef48c28fc8e3269ee2952a"
format_stamp: "Formatted at 2024-05-01 19:59:54 on dist-test-slave-z8x3"
I20240501 19:59:54.895987 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:59:54.911764 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:59:54.912896 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:59:54.914237 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:59:54.916383 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:59:54.916576 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:54.916800 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:59:54.916949 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:54.956898 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.130:35829
I20240501 19:59:54.957001 26319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.130:35829 every 8 connection(s)
I20240501 19:59:54.967123 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 19:59:54.974877 26321 heartbeater.cc:347] Connected to a master server at 127.24.69.188:34347
I20240501 19:59:54.975351 26321 heartbeater.cc:464] Registering TS with master...
I20240501 19:59:54.976215 26321 heartbeater.cc:511] Master 127.24.69.188:34347 requested a full tablet report, sending...
I20240501 19:59:54.978281 26320 heartbeater.cc:347] Connected to a master server at 127.24.69.189:45901
I20240501 19:59:54.978724 26320 heartbeater.cc:464] Registering TS with master...
I20240501 19:59:54.979091 26091 ts_manager.cc:194] Registered new tserver with Master: 58c44d190aef48c28fc8e3269ee2952a (127.24.69.130:35829)
I20240501 19:59:54.979800 26320 heartbeater.cc:511] Master 127.24.69.189:45901 requested a full tablet report, sending...
I20240501 19:59:54.983496 26026 ts_manager.cc:194] Registered new tserver with Master: 58c44d190aef48c28fc8e3269ee2952a (127.24.69.130:35829)
W20240501 19:59:54.985352 26327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:54.985703 26328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:54.985775 26322 heartbeater.cc:347] Connected to a master server at 127.24.69.190:35029
I20240501 19:59:54.986402 26322 heartbeater.cc:464] Registering TS with master...
I20240501 19:59:54.987669 26322 heartbeater.cc:511] Master 127.24.69.190:35029 requested a full tablet report, sending...
W20240501 19:59:54.988629 26330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:54.988557 26134 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:59:54.989260 26134 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7620906a374531b01c4c1a2e56c8da" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } } peers { permanent_uuid: "08ddae5019c54872b0b497b9105de9ed" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } } peers { permanent_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } }
I20240501 19:59:54.990821 26134 leader_election.cc:288] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e7620906a374531b01c4c1a2e56c8da (127.24.69.190:35029), 08ddae5019c54872b0b497b9105de9ed (127.24.69.189:45901)
I20240501 19:59:54.991174 25961 ts_manager.cc:194] Registered new tserver with Master: 58c44d190aef48c28fc8e3269ee2952a (127.24.69.130:35829)
I20240501 19:59:54.991719 25971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e7620906a374531b01c4c1a2e56c8da" is_pre_election: true
I20240501 19:59:54.992343 25971 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0844a5f3d7c4428bc1c0c10ba3a4186 in term 0.
I20240501 19:59:54.992162 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ddae5019c54872b0b497b9105de9ed" is_pre_election: true
I20240501 19:59:54.992874 26036 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0844a5f3d7c4428bc1c0c10ba3a4186 in term 0.
I20240501 19:59:54.993398 26076 leader_election.cc:302] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e7620906a374531b01c4c1a2e56c8da, d0844a5f3d7c4428bc1c0c10ba3a4186; no voters:
I20240501 19:59:54.994019 26134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:59:54.994369 26134 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:59:54.994621 26134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:54.995491 24854 server_base.cc:1034] running on GCE node
I20240501 19:59:54.996501 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:59:54.996773 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:59:54.996989 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593594996971 us; error 0 us; skew 500 ppm
I20240501 19:59:54.997682 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 19:59:55.000603 24854 webserver.cc:460] Webserver started at http://127.24.69.131:34363/ using document root <none> and password file <none>
I20240501 19:59:55.001230 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 19:59:55.001021 26134 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7620906a374531b01c4c1a2e56c8da" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } } peers { permanent_uuid: "08ddae5019c54872b0b497b9105de9ed" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } } peers { permanent_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } }
I20240501 19:59:55.001494 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:59:55.001920 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:59:55.002883 26134 leader_election.cc:288] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [CANDIDATE]: Term 1 election: Requested vote from peers 0e7620906a374531b01c4c1a2e56c8da (127.24.69.190:35029), 08ddae5019c54872b0b497b9105de9ed (127.24.69.189:45901)
I20240501 19:59:55.003247 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-2-root/instance:
uuid: "1a20d40907344ad0b789c136e8adf04a"
format_stamp: "Formatted at 2024-05-01 19:59:54 on dist-test-slave-z8x3"
I20240501 19:59:55.003557 25971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e7620906a374531b01c4c1a2e56c8da"
I20240501 19:59:55.003985 25971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:55.007731 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20240501 19:59:55.008273 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ddae5019c54872b0b497b9105de9ed"
I20240501 19:59:55.008540 25971 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0844a5f3d7c4428bc1c0c10ba3a4186 in term 1.
I20240501 19:59:55.008844 26036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:55.009433 26076 leader_election.cc:302] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e7620906a374531b01c4c1a2e56c8da, d0844a5f3d7c4428bc1c0c10ba3a4186; no voters:
I20240501 19:59:55.010004 26134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:59:55.010986 26134 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [term 1 LEADER]: Becoming Leader. State: Replica: d0844a5f3d7c4428bc1c0c10ba3a4186, State: Running, Role: LEADER
I20240501 19:59:55.011600 26134 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7620906a374531b01c4c1a2e56c8da" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } } peers { permanent_uuid: "08ddae5019c54872b0b497b9105de9ed" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } } peers { permanent_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } }
I20240501 19:59:55.012588 26336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:55.013643 24854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:59:55.014032 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-2-root
uuid: "1a20d40907344ad0b789c136e8adf04a"
format_stamp: "Formatted at 2024-05-01 19:59:54 on dist-test-slave-z8x3"
I20240501 19:59:55.014436 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593576104309-24854-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:59:55.015054 26036 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0844a5f3d7c4428bc1c0c10ba3a4186 in term 1.
I20240501 19:59:55.016700 26335 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0844a5f3d7c4428bc1c0c10ba3a4186. Latest consensus state: current_term: 1 leader_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7620906a374531b01c4c1a2e56c8da" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } } peers { permanent_uuid: "08ddae5019c54872b0b497b9105de9ed" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } } peers { permanent_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } } }
I20240501 19:59:55.017272 26335 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [sys.catalog]: This master's current role is: LEADER
I20240501 19:59:55.019222 26339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:59:55.025835 26339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:59:55.034518 25971 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da [term 1 FOLLOWER]: Refusing update from remote peer d0844a5f3d7c4428bc1c0c10ba3a4186: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:59:55.034482 26036 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [term 1 FOLLOWER]: Refusing update from remote peer d0844a5f3d7c4428bc1c0c10ba3a4186: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:59:55.035845 26134 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e7620906a374531b01c4c1a2e56c8da" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:59:55.036571 26335 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08ddae5019c54872b0b497b9105de9ed" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:59:55.044409 26137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0844a5f3d7c4428bc1c0c10ba3a4186. Latest consensus state: current_term: 1 leader_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7620906a374531b01c4c1a2e56c8da" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } } peers { permanent_uuid: "08ddae5019c54872b0b497b9105de9ed" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } } peers { permanent_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } } }
I20240501 19:59:55.045435 26137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:59:55.051982 26137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7620906a374531b01c4c1a2e56c8da" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } } peers { permanent_uuid: "08ddae5019c54872b0b497b9105de9ed" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } } peers { permanent_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } } }
I20240501 19:59:55.052795 26137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:59:55.058331 26134 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7620906a374531b01c4c1a2e56c8da" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } } peers { permanent_uuid: "08ddae5019c54872b0b497b9105de9ed" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } } peers { permanent_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } } }
I20240501 19:59:55.059043 26134 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [sys.catalog]: This master's current role is: LEADER
I20240501 19:59:55.059819 26339 catalog_manager.cc:1347] Generated new cluster ID: edb4c4f0d5534a5db799e109b9f005a9
I20240501 19:59:55.060093 26339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:59:55.061810 26143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0844a5f3d7c4428bc1c0c10ba3a4186. Latest consensus state: current_term: 1 leader_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7620906a374531b01c4c1a2e56c8da" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } } peers { permanent_uuid: "08ddae5019c54872b0b497b9105de9ed" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } } peers { permanent_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } } }
I20240501 19:59:55.062548 26143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:59:55.064738 26335 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7620906a374531b01c4c1a2e56c8da" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } } peers { permanent_uuid: "08ddae5019c54872b0b497b9105de9ed" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } } peers { permanent_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } } }
I20240501 19:59:55.065583 26335 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [sys.catalog]: This master's current role is: LEADER
I20240501 19:59:55.068214 26143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7620906a374531b01c4c1a2e56c8da" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } } peers { permanent_uuid: "08ddae5019c54872b0b497b9105de9ed" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } } peers { permanent_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } } }
I20240501 19:59:55.068961 26143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:59:55.078461 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:59:55.079910 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:59:55.081599 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:59:55.084398 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:59:55.084635 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:55.084872 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:59:55.085062 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:55.103061 26339 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:59:55.104736 26339 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:59:55.144287 26339 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186: Generated new TSK 0
I20240501 19:59:55.145038 26339 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:59:55.153136 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.131:41863
I20240501 19:59:55.153249 26410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.131:41863 every 8 connection(s)
I20240501 19:59:55.174191 26411 heartbeater.cc:347] Connected to a master server at 127.24.69.189:45901
I20240501 19:59:55.174572 26411 heartbeater.cc:464] Registering TS with master...
I20240501 19:59:55.175276 26412 heartbeater.cc:347] Connected to a master server at 127.24.69.188:34347
I20240501 19:59:55.175532 26411 heartbeater.cc:511] Master 127.24.69.189:45901 requested a full tablet report, sending...
I20240501 19:59:55.175630 26412 heartbeater.cc:464] Registering TS with master...
I20240501 19:59:55.176528 26412 heartbeater.cc:511] Master 127.24.69.188:34347 requested a full tablet report, sending...
I20240501 19:59:55.178416 26091 ts_manager.cc:194] Registered new tserver with Master: 1a20d40907344ad0b789c136e8adf04a (127.24.69.131:41863)
I20240501 19:59:55.179486 26026 ts_manager.cc:194] Registered new tserver with Master: 1a20d40907344ad0b789c136e8adf04a (127.24.69.131:41863)
I20240501 19:59:55.180224 26091 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46208
I20240501 19:59:55.180785 26413 heartbeater.cc:347] Connected to a master server at 127.24.69.190:35029
I20240501 19:59:55.181171 26413 heartbeater.cc:464] Registering TS with master...
I20240501 19:59:55.181922 26413 heartbeater.cc:511] Master 127.24.69.190:35029 requested a full tablet report, sending...
I20240501 19:59:55.183930 25961 ts_manager.cc:194] Registered new tserver with Master: 1a20d40907344ad0b789c136e8adf04a (127.24.69.131:41863)
I20240501 19:59:55.184041 24854 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.021610812s
I20240501 19:59:55.765547 26164 catalog_manager.cc:1259] Loaded cluster ID: edb4c4f0d5534a5db799e109b9f005a9
I20240501 19:59:55.765936 26164 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da: loading cluster ID for follower catalog manager: success
I20240501 19:59:55.769196 26170 catalog_manager.cc:1259] Loaded cluster ID: edb4c4f0d5534a5db799e109b9f005a9
I20240501 19:59:55.769447 26170 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed: loading cluster ID for follower catalog manager: success
I20240501 19:59:55.770359 26164 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da: acquiring CA information for follower catalog manager: success
I20240501 19:59:55.773545 26164 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 19:59:55.773598 26170 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed: acquiring CA information for follower catalog manager: success
I20240501 19:59:55.776661 26170 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 19:59:55.876040 26091 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46178
I20240501 19:59:55.983570 26091 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46194
I20240501 19:59:56.182626 26412 heartbeater.cc:503] Master 127.24.69.188:34347 was elected leader, sending a full tablet report...
I20240501 19:59:56.747352 26150 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:59:56.748986 24854 test_util.cc:268] Using random seed: -297487275
I20240501 19:59:56.790195 26091 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46212:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:59:56.868482 26207 tablet_service.cc:1467] Processing CreateTablet for tablet b16f501c76d44b0e9457adc4c8b6da4b (DEFAULT_TABLE table=test-workload [id=6abdad9d65344f5a9f551a5931cde687]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240501 19:59:56.869136 26285 tablet_service.cc:1467] Processing CreateTablet for tablet b40b7f23b40e4a14be04a994c31e6484 (DEFAULT_TABLE table=test-workload [id=6abdad9d65344f5a9f551a5931cde687]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240501 19:59:56.870102 26207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b16f501c76d44b0e9457adc4c8b6da4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:56.870775 26285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40b7f23b40e4a14be04a994c31e6484. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:56.870766 26376 tablet_service.cc:1467] Processing CreateTablet for tablet d3d18c9579484760bfaf052f66a644d1 (DEFAULT_TABLE table=test-workload [id=6abdad9d65344f5a9f551a5931cde687]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240501 19:59:56.872382 26376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3d18c9579484760bfaf052f66a644d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:56.878744 26243 heartbeater.cc:503] Master 127.24.69.188:34347 was elected leader, sending a full tablet report...
I20240501 19:59:56.885794 26447 tablet_bootstrap.cc:492] T b16f501c76d44b0e9457adc4c8b6da4b P 88ecb0a11a8a4950a8550193a3f81d9f: Bootstrap starting.
I20240501 19:59:56.891927 26447 tablet_bootstrap.cc:654] T b16f501c76d44b0e9457adc4c8b6da4b P 88ecb0a11a8a4950a8550193a3f81d9f: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:56.893326 26448 tablet_bootstrap.cc:492] T b40b7f23b40e4a14be04a994c31e6484 P 58c44d190aef48c28fc8e3269ee2952a: Bootstrap starting.
I20240501 19:59:56.895068 26449 tablet_bootstrap.cc:492] T d3d18c9579484760bfaf052f66a644d1 P 1a20d40907344ad0b789c136e8adf04a: Bootstrap starting.
I20240501 19:59:56.900527 26449 tablet_bootstrap.cc:654] T d3d18c9579484760bfaf052f66a644d1 P 1a20d40907344ad0b789c136e8adf04a: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:56.901672 26448 tablet_bootstrap.cc:654] T b40b7f23b40e4a14be04a994c31e6484 P 58c44d190aef48c28fc8e3269ee2952a: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:56.904374 26447 tablet_bootstrap.cc:492] T b16f501c76d44b0e9457adc4c8b6da4b P 88ecb0a11a8a4950a8550193a3f81d9f: No bootstrap required, opened a new log
I20240501 19:59:56.904963 26447 ts_tablet_manager.cc:1367] T b16f501c76d44b0e9457adc4c8b6da4b P 88ecb0a11a8a4950a8550193a3f81d9f: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.011s
I20240501 19:59:56.905836 26449 tablet_bootstrap.cc:492] T d3d18c9579484760bfaf052f66a644d1 P 1a20d40907344ad0b789c136e8adf04a: No bootstrap required, opened a new log
I20240501 19:59:56.906267 26449 ts_tablet_manager.cc:1367] T d3d18c9579484760bfaf052f66a644d1 P 1a20d40907344ad0b789c136e8adf04a: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240501 19:59:56.907496 26448 tablet_bootstrap.cc:492] T b40b7f23b40e4a14be04a994c31e6484 P 58c44d190aef48c28fc8e3269ee2952a: No bootstrap required, opened a new log
I20240501 19:59:56.907960 26448 ts_tablet_manager.cc:1367] T b40b7f23b40e4a14be04a994c31e6484 P 58c44d190aef48c28fc8e3269ee2952a: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20240501 19:59:56.907759 26447 raft_consensus.cc:348] T b16f501c76d44b0e9457adc4c8b6da4b P 88ecb0a11a8a4950a8550193a3f81d9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ecb0a11a8a4950a8550193a3f81d9f" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 46143 } }
I20240501 19:59:56.908250 26449 raft_consensus.cc:348] T d3d18c9579484760bfaf052f66a644d1 P 1a20d40907344ad0b789c136e8adf04a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a20d40907344ad0b789c136e8adf04a" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 41863 } }
I20240501 19:59:56.908541 26447 raft_consensus.cc:374] T b16f501c76d44b0e9457adc4c8b6da4b P 88ecb0a11a8a4950a8550193a3f81d9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:56.908812 26449 raft_consensus.cc:374] T d3d18c9579484760bfaf052f66a644d1 P 1a20d40907344ad0b789c136e8adf04a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:56.908977 26447 raft_consensus.cc:729] T b16f501c76d44b0e9457adc4c8b6da4b P 88ecb0a11a8a4950a8550193a3f81d9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88ecb0a11a8a4950a8550193a3f81d9f, State: Initialized, Role: FOLLOWER
I20240501 19:59:56.909168 26449 raft_consensus.cc:729] T d3d18c9579484760bfaf052f66a644d1 P 1a20d40907344ad0b789c136e8adf04a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a20d40907344ad0b789c136e8adf04a, State: Initialized, Role: FOLLOWER
I20240501 19:59:56.909921 26449 consensus_queue.cc:260] T d3d18c9579484760bfaf052f66a644d1 P 1a20d40907344ad0b789c136e8adf04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a20d40907344ad0b789c136e8adf04a" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 41863 } }
I20240501 19:59:56.909940 26447 consensus_queue.cc:260] T b16f501c76d44b0e9457adc4c8b6da4b P 88ecb0a11a8a4950a8550193a3f81d9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ecb0a11a8a4950a8550193a3f81d9f" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 46143 } }
I20240501 19:59:56.910593 26449 raft_consensus.cc:388] T d3d18c9579484760bfaf052f66a644d1 P 1a20d40907344ad0b789c136e8adf04a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:59:56.910815 26447 raft_consensus.cc:388] T b16f501c76d44b0e9457adc4c8b6da4b P 88ecb0a11a8a4950a8550193a3f81d9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:59:56.910976 26449 raft_consensus.cc:482] T d3d18c9579484760bfaf052f66a644d1 P 1a20d40907344ad0b789c136e8adf04a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:59:56.910698 26448 raft_consensus.cc:348] T b40b7f23b40e4a14be04a994c31e6484 P 58c44d190aef48c28fc8e3269ee2952a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c44d190aef48c28fc8e3269ee2952a" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 35829 } }
I20240501 19:59:56.911290 26447 raft_consensus.cc:482] T b16f501c76d44b0e9457adc4c8b6da4b P 88ecb0a11a8a4950a8550193a3f81d9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:59:56.911432 26449 raft_consensus.cc:3037] T d3d18c9579484760bfaf052f66a644d1 P 1a20d40907344ad0b789c136e8adf04a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:56.911515 26448 raft_consensus.cc:374] T b40b7f23b40e4a14be04a994c31e6484 P 58c44d190aef48c28fc8e3269ee2952a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:56.911692 26447 raft_consensus.cc:3037] T b16f501c76d44b0e9457adc4c8b6da4b P 88ecb0a11a8a4950a8550193a3f81d9f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:56.911960 26448 raft_consensus.cc:729] T b40b7f23b40e4a14be04a994c31e6484 P 58c44d190aef48c28fc8e3269ee2952a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58c44d190aef48c28fc8e3269ee2952a, State: Initialized, Role: FOLLOWER
I20240501 19:59:56.912690 26448 consensus_queue.cc:260] T b40b7f23b40e4a14be04a994c31e6484 P 58c44d190aef48c28fc8e3269ee2952a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c44d190aef48c28fc8e3269ee2952a" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 35829 } }
I20240501 19:59:56.913421 26448 raft_consensus.cc:388] T b40b7f23b40e4a14be04a994c31e6484 P 58c44d190aef48c28fc8e3269ee2952a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:59:56.913774 26448 raft_consensus.cc:482] T b40b7f23b40e4a14be04a994c31e6484 P 58c44d190aef48c28fc8e3269ee2952a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:59:56.914171 26448 raft_consensus.cc:3037] T b40b7f23b40e4a14be04a994c31e6484 P 58c44d190aef48c28fc8e3269ee2952a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:56.917585 26449 raft_consensus.cc:504] T d3d18c9579484760bfaf052f66a644d1 P 1a20d40907344ad0b789c136e8adf04a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a20d40907344ad0b789c136e8adf04a" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 41863 } }
I20240501 19:59:56.918182 26449 leader_election.cc:302] T d3d18c9579484760bfaf052f66a644d1 P 1a20d40907344ad0b789c136e8adf04a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a20d40907344ad0b789c136e8adf04a; no voters:
I20240501 19:59:56.918378 26447 raft_consensus.cc:504] T b16f501c76d44b0e9457adc4c8b6da4b P 88ecb0a11a8a4950a8550193a3f81d9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ecb0a11a8a4950a8550193a3f81d9f" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 46143 } }
I20240501 19:59:56.919137 26447 leader_election.cc:302] T b16f501c76d44b0e9457adc4c8b6da4b P 88ecb0a11a8a4950a8550193a3f81d9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88ecb0a11a8a4950a8550193a3f81d9f; no voters:
I20240501 19:59:56.919581 26449 leader_election.cc:288] T d3d18c9579484760bfaf052f66a644d1 P 1a20d40907344ad0b789c136e8adf04a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:59:56.920051 26453 raft_consensus.cc:2781] T d3d18c9579484760bfaf052f66a644d1 P 1a20d40907344ad0b789c136e8adf04a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:59:56.920907 26448 raft_consensus.cc:504] T b40b7f23b40e4a14be04a994c31e6484 P 58c44d190aef48c28fc8e3269ee2952a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c44d190aef48c28fc8e3269ee2952a" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 35829 } }
I20240501 19:59:56.921938 26448 leader_election.cc:302] T b40b7f23b40e4a14be04a994c31e6484 P 58c44d190aef48c28fc8e3269ee2952a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58c44d190aef48c28fc8e3269ee2952a; no voters:
I20240501 19:59:56.925870 26457 raft_consensus.cc:2781] T b40b7f23b40e4a14be04a994c31e6484 P 58c44d190aef48c28fc8e3269ee2952a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:59:56.926389 26448 leader_election.cc:288] T b40b7f23b40e4a14be04a994c31e6484 P 58c44d190aef48c28fc8e3269ee2952a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:59:56.927232 26454 raft_consensus.cc:2781] T b16f501c76d44b0e9457adc4c8b6da4b P 88ecb0a11a8a4950a8550193a3f81d9f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:59:56.927784 26447 leader_election.cc:288] T b16f501c76d44b0e9457adc4c8b6da4b P 88ecb0a11a8a4950a8550193a3f81d9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:59:56.930007 26448 ts_tablet_manager.cc:1398] T b40b7f23b40e4a14be04a994c31e6484 P 58c44d190aef48c28fc8e3269ee2952a: Time spent starting tablet: real 0.022s user 0.010s sys 0.007s
I20240501 19:59:56.931555 26321 heartbeater.cc:503] Master 127.24.69.188:34347 was elected leader, sending a full tablet report...
I20240501 19:59:56.932299 26449 ts_tablet_manager.cc:1398] T d3d18c9579484760bfaf052f66a644d1 P 1a20d40907344ad0b789c136e8adf04a: Time spent starting tablet: real 0.026s user 0.023s sys 0.004s
I20240501 19:59:56.933302 26453 raft_consensus.cc:686] T d3d18c9579484760bfaf052f66a644d1 P 1a20d40907344ad0b789c136e8adf04a [term 1 LEADER]: Becoming Leader. State: Replica: 1a20d40907344ad0b789c136e8adf04a, State: Running, Role: LEADER
I20240501 19:59:56.934237 26453 consensus_queue.cc:237] T d3d18c9579484760bfaf052f66a644d1 P 1a20d40907344ad0b789c136e8adf04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a20d40907344ad0b789c136e8adf04a" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 41863 } }
I20240501 19:59:56.938251 26447 ts_tablet_manager.cc:1398] T b16f501c76d44b0e9457adc4c8b6da4b P 88ecb0a11a8a4950a8550193a3f81d9f: Time spent starting tablet: real 0.033s user 0.014s sys 0.011s
I20240501 19:59:56.936792 26457 raft_consensus.cc:686] T b40b7f23b40e4a14be04a994c31e6484 P 58c44d190aef48c28fc8e3269ee2952a [term 1 LEADER]: Becoming Leader. State: Replica: 58c44d190aef48c28fc8e3269ee2952a, State: Running, Role: LEADER
I20240501 19:59:56.940167 26454 raft_consensus.cc:686] T b16f501c76d44b0e9457adc4c8b6da4b P 88ecb0a11a8a4950a8550193a3f81d9f [term 1 LEADER]: Becoming Leader. State: Replica: 88ecb0a11a8a4950a8550193a3f81d9f, State: Running, Role: LEADER
I20240501 19:59:56.941133 26454 consensus_queue.cc:237] T b16f501c76d44b0e9457adc4c8b6da4b P 88ecb0a11a8a4950a8550193a3f81d9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ecb0a11a8a4950a8550193a3f81d9f" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 46143 } }
I20240501 19:59:56.943035 26457 consensus_queue.cc:237] T b40b7f23b40e4a14be04a994c31e6484 P 58c44d190aef48c28fc8e3269ee2952a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c44d190aef48c28fc8e3269ee2952a" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 35829 } }
I20240501 19:59:56.946724 26091 catalog_manager.cc:5484] T d3d18c9579484760bfaf052f66a644d1 P 1a20d40907344ad0b789c136e8adf04a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a20d40907344ad0b789c136e8adf04a (127.24.69.131). New cstate: current_term: 1 leader_uuid: "1a20d40907344ad0b789c136e8adf04a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a20d40907344ad0b789c136e8adf04a" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 41863 } health_report { overall_health: HEALTHY } } }
I20240501 19:59:56.953835 26090 catalog_manager.cc:5484] T b16f501c76d44b0e9457adc4c8b6da4b P 88ecb0a11a8a4950a8550193a3f81d9f reported cstate change: term changed from 0 to 1, leader changed from <none> to 88ecb0a11a8a4950a8550193a3f81d9f (127.24.69.129). New cstate: current_term: 1 leader_uuid: "88ecb0a11a8a4950a8550193a3f81d9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ecb0a11a8a4950a8550193a3f81d9f" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 46143 } health_report { overall_health: HEALTHY } } }
I20240501 19:59:56.981848 26090 catalog_manager.cc:5484] T b40b7f23b40e4a14be04a994c31e6484 P 58c44d190aef48c28fc8e3269ee2952a reported cstate change: term changed from 0 to 1, leader changed from <none> to 58c44d190aef48c28fc8e3269ee2952a (127.24.69.130). New cstate: current_term: 1 leader_uuid: "58c44d190aef48c28fc8e3269ee2952a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c44d190aef48c28fc8e3269ee2952a" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 35829 } health_report { overall_health: HEALTHY } } }
I20240501 19:59:57.021422 24854 master.cc:537] Master@127.24.69.188:34347 shutting down...
I20240501 19:59:57.044642 24854 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:59:57.045442 24854 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:57.045750 24854 tablet_replica.cc:330] T 00000000000000000000000000000000 P d0844a5f3d7c4428bc1c0c10ba3a4186: stopping tablet replica
I20240501 19:59:57.065830 24854 master.cc:552] Master@127.24.69.188:34347 shutdown complete.
W20240501 19:59:57.976711 26412 heartbeater.cc:644] Failed to heartbeat to 127.24.69.188:34347 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.69.188:34347: connect: Connection refused (error 111)
I20240501 19:59:58.498481 26471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0844a5f3d7c4428bc1c0c10ba3a4186)
I20240501 19:59:58.498916 26471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7620906a374531b01c4c1a2e56c8da" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } } peers { permanent_uuid: "08ddae5019c54872b0b497b9105de9ed" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } } peers { permanent_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } }
I20240501 19:59:58.500504 26471 leader_election.cc:288] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08ddae5019c54872b0b497b9105de9ed (127.24.69.189:45901), d0844a5f3d7c4428bc1c0c10ba3a4186 (127.24.69.188:34347)
I20240501 19:59:58.501420 26036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0e7620906a374531b01c4c1a2e56c8da" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "08ddae5019c54872b0b497b9105de9ed" is_pre_election: true
W20240501 19:59:58.503039 25945 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.69.188:34347: connect: Connection refused (error 111) [suppressed 25 similar messages]
I20240501 19:59:58.504174 26473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0844a5f3d7c4428bc1c0c10ba3a4186)
I20240501 19:59:58.504623 26473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7620906a374531b01c4c1a2e56c8da" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } } peers { permanent_uuid: "08ddae5019c54872b0b497b9105de9ed" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } } peers { permanent_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } }
I20240501 19:59:58.506315 26473 leader_election.cc:288] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e7620906a374531b01c4c1a2e56c8da (127.24.69.190:35029), d0844a5f3d7c4428bc1c0c10ba3a4186 (127.24.69.188:34347)
W20240501 19:59:58.506523 25945 leader_election.cc:334] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d0844a5f3d7c4428bc1c0c10ba3a4186 (127.24.69.188:34347): Network error: Client connection negotiation failed: client connection to 127.24.69.188:34347: connect: Connection refused (error 111)
I20240501 19:59:58.506999 25945 leader_election.cc:302] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e7620906a374531b01c4c1a2e56c8da; no voters: 08ddae5019c54872b0b497b9105de9ed, d0844a5f3d7c4428bc1c0c10ba3a4186
I20240501 19:59:58.507308 25970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08ddae5019c54872b0b497b9105de9ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "0e7620906a374531b01c4c1a2e56c8da" is_pre_election: true
I20240501 19:59:58.507972 25970 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08ddae5019c54872b0b497b9105de9ed in term 1.
I20240501 19:59:58.508432 26471 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:59:58.509106 26012 leader_election.cc:302] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08ddae5019c54872b0b497b9105de9ed, 0e7620906a374531b01c4c1a2e56c8da; no voters:
I20240501 19:59:58.509809 26473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:59:58.510144 26473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader d0844a5f3d7c4428bc1c0c10ba3a4186)
I20240501 19:59:58.510407 26473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:59:58.511662 26010 leader_election.cc:334] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d0844a5f3d7c4428bc1c0c10ba3a4186 (127.24.69.188:34347): Network error: Client connection negotiation failed: client connection to 127.24.69.188:34347: connect: Connection refused (error 111)
I20240501 19:59:58.515434 26473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7620906a374531b01c4c1a2e56c8da" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } } peers { permanent_uuid: "08ddae5019c54872b0b497b9105de9ed" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } } peers { permanent_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } }
I20240501 19:59:58.516894 26473 leader_election.cc:288] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [CANDIDATE]: Term 2 election: Requested vote from peers 0e7620906a374531b01c4c1a2e56c8da (127.24.69.190:35029), d0844a5f3d7c4428bc1c0c10ba3a4186 (127.24.69.188:34347)
I20240501 19:59:58.517485 25970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08ddae5019c54872b0b497b9105de9ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "0e7620906a374531b01c4c1a2e56c8da"
I20240501 19:59:58.517915 25970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:59:58.519821 26010 leader_election.cc:334] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d0844a5f3d7c4428bc1c0c10ba3a4186 (127.24.69.188:34347): Network error: Client connection negotiation failed: client connection to 127.24.69.188:34347: connect: Connection refused (error 111)
I20240501 19:59:58.522351 25970 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08ddae5019c54872b0b497b9105de9ed in term 2.
I20240501 19:59:58.523097 26012 leader_election.cc:302] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08ddae5019c54872b0b497b9105de9ed, 0e7620906a374531b01c4c1a2e56c8da; no voters: d0844a5f3d7c4428bc1c0c10ba3a4186
I20240501 19:59:58.523707 26473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:59:58.524498 26473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [term 2 LEADER]: Becoming Leader. State: Replica: 08ddae5019c54872b0b497b9105de9ed, State: Running, Role: LEADER
I20240501 19:59:58.525070 26473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7620906a374531b01c4c1a2e56c8da" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } } peers { permanent_uuid: "08ddae5019c54872b0b497b9105de9ed" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } } peers { permanent_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } }
I20240501 19:59:58.527779 26477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08ddae5019c54872b0b497b9105de9ed. Latest consensus state: current_term: 2 leader_uuid: "08ddae5019c54872b0b497b9105de9ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7620906a374531b01c4c1a2e56c8da" member_type: VOTER last_known_addr { host: "127.24.69.190" port: 35029 } } peers { permanent_uuid: "08ddae5019c54872b0b497b9105de9ed" member_type: VOTER last_known_addr { host: "127.24.69.189" port: 45901 } } peers { permanent_uuid: "d0844a5f3d7c4428bc1c0c10ba3a4186" member_type: VOTER last_known_addr { host: "127.24.69.188" port: 34347 } } }
I20240501 19:59:58.528398 26477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [sys.catalog]: This master's current role is: LEADER
I20240501 19:59:58.529706 26479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:59:58.534689 26479 catalog_manager.cc:669] Loaded metadata for table test-workload [id=6abdad9d65344f5a9f551a5931cde687]
I20240501 19:59:58.539363 26479 tablet_loader.cc:96] loaded metadata for tablet b16f501c76d44b0e9457adc4c8b6da4b (table test-workload [id=6abdad9d65344f5a9f551a5931cde687])
I20240501 19:59:58.540445 26479 tablet_loader.cc:96] loaded metadata for tablet b40b7f23b40e4a14be04a994c31e6484 (table test-workload [id=6abdad9d65344f5a9f551a5931cde687])
I20240501 19:59:58.541245 26479 tablet_loader.cc:96] loaded metadata for tablet d3d18c9579484760bfaf052f66a644d1 (table test-workload [id=6abdad9d65344f5a9f551a5931cde687])
I20240501 19:59:58.542044 26479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:59:58.544981 26479 catalog_manager.cc:1259] Loaded cluster ID: edb4c4f0d5534a5db799e109b9f005a9
I20240501 19:59:58.545174 26479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:59:58.548373 26479 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:59:58.551285 26479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed: Loaded TSK: 0
I20240501 19:59:58.552090 26479 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:59:58.763944 26169 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:59:58.764245 26169 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:59:58.858004 24854 tablet_server.cc:178] TabletServer@127.24.69.129:0 shutting down...
I20240501 19:59:58.881148 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:59:58.881827 24854 tablet_replica.cc:330] T b16f501c76d44b0e9457adc4c8b6da4b P 88ecb0a11a8a4950a8550193a3f81d9f: stopping tablet replica
I20240501 19:59:58.882378 24854 raft_consensus.cc:2229] T b16f501c76d44b0e9457adc4c8b6da4b P 88ecb0a11a8a4950a8550193a3f81d9f [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:59:58.882763 24854 raft_consensus.cc:2258] T b16f501c76d44b0e9457adc4c8b6da4b P 88ecb0a11a8a4950a8550193a3f81d9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:58.899897 24854 tablet_server.cc:194] TabletServer@127.24.69.129:0 shutdown complete.
I20240501 19:59:58.907891 24854 tablet_server.cc:178] TabletServer@127.24.69.130:0 shutting down...
I20240501 19:59:58.927763 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:59:58.928402 24854 tablet_replica.cc:330] T b40b7f23b40e4a14be04a994c31e6484 P 58c44d190aef48c28fc8e3269ee2952a: stopping tablet replica
I20240501 19:59:58.928867 24854 raft_consensus.cc:2229] T b40b7f23b40e4a14be04a994c31e6484 P 58c44d190aef48c28fc8e3269ee2952a [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:59:58.929246 24854 raft_consensus.cc:2258] T b40b7f23b40e4a14be04a994c31e6484 P 58c44d190aef48c28fc8e3269ee2952a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:58.946435 24854 tablet_server.cc:194] TabletServer@127.24.69.130:0 shutdown complete.
I20240501 19:59:58.954439 24854 tablet_server.cc:178] TabletServer@127.24.69.131:0 shutting down...
I20240501 19:59:58.977300 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:59:58.977924 24854 tablet_replica.cc:330] T d3d18c9579484760bfaf052f66a644d1 P 1a20d40907344ad0b789c136e8adf04a: stopping tablet replica
I20240501 19:59:58.978403 24854 raft_consensus.cc:2229] T d3d18c9579484760bfaf052f66a644d1 P 1a20d40907344ad0b789c136e8adf04a [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:59:58.978785 24854 raft_consensus.cc:2258] T d3d18c9579484760bfaf052f66a644d1 P 1a20d40907344ad0b789c136e8adf04a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:58.996078 24854 tablet_server.cc:194] TabletServer@127.24.69.131:0 shutdown complete.
I20240501 19:59:59.004338 24854 master.cc:537] Master@127.24.69.190:35029 shutting down...
I20240501 19:59:59.022361 24854 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:59:59.022890 24854 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:59.023171 24854 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0e7620906a374531b01c4c1a2e56c8da: stopping tablet replica
W20240501 19:59:59.040639 26010 consensus_peers.cc:485] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed -> Peer d0844a5f3d7c4428bc1c0c10ba3a4186 (127.24.69.188:34347): Couldn't send request to peer d0844a5f3d7c4428bc1c0c10ba3a4186. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.188:34347: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:59:59.041203 24854 master.cc:552] Master@127.24.69.190:35029 shutdown complete.
I20240501 19:59:59.052997 24854 master.cc:537] Master@127.24.69.189:45901 shutting down...
I20240501 19:59:59.069901 24854 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [term 2 LEADER]: Raft consensus shutting down.
I20240501 19:59:59.070555 24854 pending_rounds.cc:62] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed: Trying to abort 1 pending ops.
I20240501 19:59:59.070749 24854 pending_rounds.cc:68] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed: Aborting op as it isn't in flight: id { term: 2 index: 10 } timestamp: 7022975379564445696 op_type: NO_OP noop_request { }
I20240501 19:59:59.071036 24854 raft_consensus.cc:2866] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240501 19:59:59.071250 24854 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:59:59.071475 24854 tablet_replica.cc:330] T 00000000000000000000000000000000 P 08ddae5019c54872b0b497b9105de9ed: stopping tablet replica
I20240501 19:59:59.088915 24854 master.cc:552] Master@127.24.69.189:45901 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (4747 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240501 19:59:59.118983 24854 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.69.190:39931
I20240501 19:59:59.119946 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:59:59.125589 26483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:59.126341 26484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:59.126914 26486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:59.127521 24854 server_base.cc:1034] running on GCE node
I20240501 19:59:59.128204 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:59:59.128432 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:59:59.128551 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593599128538 us; error 0 us; skew 500 ppm
I20240501 19:59:59.128981 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 19:59:59.131290 24854 webserver.cc:460] Webserver started at http://127.24.69.190:42521/ using document root <none> and password file <none>
I20240501 19:59:59.131676 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 19:59:59.131821 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:59:59.132035 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:59:59.134662 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593576104309-24854-0/minicluster-data/master-0-root/instance:
uuid: "69d9e669a0d14218b496566a27f0a70d"
format_stamp: "Formatted at 2024-05-01 19:59:59 on dist-test-slave-z8x3"
I20240501 19:59:59.138743 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20240501 19:59:59.141764 26491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:59.142591 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:59:59.142872 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593576104309-24854-0/minicluster-data/master-0-root
uuid: "69d9e669a0d14218b496566a27f0a70d"
format_stamp: "Formatted at 2024-05-01 19:59:59 on dist-test-slave-z8x3"
I20240501 19:59:59.143114 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593576104309-24854-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593576104309-24854-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593576104309-24854-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:59:59.167423 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:59:59.168561 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:59:59.203785 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.190:39931
I20240501 19:59:59.203893 26542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.190:39931 every 8 connection(s)
I20240501 19:59:59.207378 26543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:59:59.216431 26543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69d9e669a0d14218b496566a27f0a70d: Bootstrap starting.
I20240501 19:59:59.220485 26543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69d9e669a0d14218b496566a27f0a70d: Neither blocks nor log segments found. Creating new log.
I20240501 19:59:59.224151 26543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69d9e669a0d14218b496566a27f0a70d: No bootstrap required, opened a new log
I20240501 19:59:59.225946 26543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69d9e669a0d14218b496566a27f0a70d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d9e669a0d14218b496566a27f0a70d" member_type: VOTER }
I20240501 19:59:59.226313 26543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69d9e669a0d14218b496566a27f0a70d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:59:59.226536 26543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69d9e669a0d14218b496566a27f0a70d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69d9e669a0d14218b496566a27f0a70d, State: Initialized, Role: FOLLOWER
I20240501 19:59:59.227042 26543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69d9e669a0d14218b496566a27f0a70d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d9e669a0d14218b496566a27f0a70d" member_type: VOTER }
I20240501 19:59:59.227489 26543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69d9e669a0d14218b496566a27f0a70d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:59:59.227694 26543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69d9e669a0d14218b496566a27f0a70d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:59:59.227948 26543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69d9e669a0d14218b496566a27f0a70d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:59:59.231707 26543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69d9e669a0d14218b496566a27f0a70d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d9e669a0d14218b496566a27f0a70d" member_type: VOTER }
I20240501 19:59:59.232314 26543 leader_election.cc:302] T 00000000000000000000000000000000 P 69d9e669a0d14218b496566a27f0a70d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69d9e669a0d14218b496566a27f0a70d; no voters:
I20240501 19:59:59.233358 26543 leader_election.cc:288] T 00000000000000000000000000000000 P 69d9e669a0d14218b496566a27f0a70d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:59:59.233639 26546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69d9e669a0d14218b496566a27f0a70d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:59:59.234964 26546 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69d9e669a0d14218b496566a27f0a70d [term 1 LEADER]: Becoming Leader. State: Replica: 69d9e669a0d14218b496566a27f0a70d, State: Running, Role: LEADER
I20240501 19:59:59.235574 26546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69d9e669a0d14218b496566a27f0a70d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d9e669a0d14218b496566a27f0a70d" member_type: VOTER }
I20240501 19:59:59.236116 26543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69d9e669a0d14218b496566a27f0a70d [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:59:59.238256 26548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69d9e669a0d14218b496566a27f0a70d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69d9e669a0d14218b496566a27f0a70d. Latest consensus state: current_term: 1 leader_uuid: "69d9e669a0d14218b496566a27f0a70d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d9e669a0d14218b496566a27f0a70d" member_type: VOTER } }
I20240501 19:59:59.238237 26547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69d9e669a0d14218b496566a27f0a70d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69d9e669a0d14218b496566a27f0a70d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d9e669a0d14218b496566a27f0a70d" member_type: VOTER } }
I20240501 19:59:59.238942 26548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69d9e669a0d14218b496566a27f0a70d [sys.catalog]: This master's current role is: LEADER
I20240501 19:59:59.239041 26547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69d9e669a0d14218b496566a27f0a70d [sys.catalog]: This master's current role is: LEADER
I20240501 19:59:59.242460 26553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:59:59.250020 26553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:59:59.250921 24854 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 19:59:59.257653 26553 catalog_manager.cc:1347] Generated new cluster ID: 47f95f496a404b2593e535f54cd85e82
I20240501 19:59:59.257918 26553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:59:59.278174 26553 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:59:59.279341 26553 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:59:59.289726 26553 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 69d9e669a0d14218b496566a27f0a70d: Generated new TSK 0
I20240501 19:59:59.290289 26553 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:59:59.317133 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:59:59.323007 26564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:59.324522 26565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:59.325582 26567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:59.326606 24854 server_base.cc:1034] running on GCE node
I20240501 19:59:59.327279 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:59:59.327497 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:59:59.327649 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593599327628 us; error 0 us; skew 500 ppm
I20240501 19:59:59.328117 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 19:59:59.330292 24854 webserver.cc:460] Webserver started at http://127.24.69.129:38645/ using document root <none> and password file <none>
I20240501 19:59:59.330725 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 19:59:59.330900 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:59:59.331142 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:59:59.332165 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593576104309-24854-0/minicluster-data/ts-0-root/instance:
uuid: "5f3e6e0ea51648bb90f831efd3bdd8de"
format_stamp: "Formatted at 2024-05-01 19:59:59 on dist-test-slave-z8x3"
I20240501 19:59:59.336974 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240501 19:59:59.339931 26572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:59.340668 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 19:59:59.340929 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593576104309-24854-0/minicluster-data/ts-0-root
uuid: "5f3e6e0ea51648bb90f831efd3bdd8de"
format_stamp: "Formatted at 2024-05-01 19:59:59 on dist-test-slave-z8x3"
I20240501 19:59:59.341181 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593576104309-24854-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593576104309-24854-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593576104309-24854-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:59:59.376294 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:59:59.377374 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:59:59.378775 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:59:59.380973 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:59:59.381161 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:59.381397 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:59:59.381561 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:59.420589 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.129:34497
I20240501 19:59:59.420677 26634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.129:34497 every 8 connection(s)
I20240501 19:59:59.425431 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:59:59.433125 26639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:59.434329 26640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:59.436357 26642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:59.437289 24854 server_base.cc:1034] running on GCE node
I20240501 19:59:59.438035 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:59:59.438282 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:59:59.438468 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593599438450 us; error 0 us; skew 500 ppm
I20240501 19:59:59.439056 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 19:59:59.442032 24854 webserver.cc:460] Webserver started at http://127.24.69.130:39151/ using document root <none> and password file <none>
I20240501 19:59:59.442322 26635 heartbeater.cc:347] Connected to a master server at 127.24.69.190:39931
I20240501 19:59:59.442598 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 19:59:59.442730 26635 heartbeater.cc:464] Registering TS with master...
I20240501 19:59:59.442870 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:59:59.443243 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:59:59.443679 26635 heartbeater.cc:511] Master 127.24.69.190:39931 requested a full tablet report, sending...
I20240501 19:59:59.444656 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593576104309-24854-0/minicluster-data/ts-1-root/instance:
uuid: "ab3a5fc4685a45e592318abadee052df"
format_stamp: "Formatted at 2024-05-01 19:59:59 on dist-test-slave-z8x3"
I20240501 19:59:59.445868 26508 ts_manager.cc:194] Registered new tserver with Master: 5f3e6e0ea51648bb90f831efd3bdd8de (127.24.69.129:34497)
I20240501 19:59:59.447463 26508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41334
I20240501 19:59:59.449143 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240501 19:59:59.452243 26647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:59.452973 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 19:59:59.453243 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593576104309-24854-0/minicluster-data/ts-1-root
uuid: "ab3a5fc4685a45e592318abadee052df"
format_stamp: "Formatted at 2024-05-01 19:59:59 on dist-test-slave-z8x3"
I20240501 19:59:59.453487 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593576104309-24854-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593576104309-24854-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593576104309-24854-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:59:59.464953 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:59:59.466050 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:59:59.467360 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:59:59.469483 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:59:59.469676 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:59.469889 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:59:59.470037 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:59.508536 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.130:33453
I20240501 19:59:59.508644 26709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.130:33453 every 8 connection(s)
I20240501 19:59:59.513056 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:59:59.519883 26713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:59.520855 26714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:59:59.522221 26716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:59:59.524222 24854 server_base.cc:1034] running on GCE node
I20240501 19:59:59.524935 26710 heartbeater.cc:347] Connected to a master server at 127.24.69.190:39931
I20240501 19:59:59.525126 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240501 19:59:59.525261 26710 heartbeater.cc:464] Registering TS with master...
W20240501 19:59:59.525390 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:59:59.525620 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593599525602 us; error 0 us; skew 500 ppm
I20240501 19:59:59.526059 26710 heartbeater.cc:511] Master 127.24.69.190:39931 requested a full tablet report, sending...
I20240501 19:59:59.526193 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 19:59:59.528190 26508 ts_manager.cc:194] Registered new tserver with Master: ab3a5fc4685a45e592318abadee052df (127.24.69.130:33453)
I20240501 19:59:59.529114 24854 webserver.cc:460] Webserver started at http://127.24.69.131:39569/ using document root <none> and password file <none>
I20240501 19:59:59.529727 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 19:59:59.529954 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:59:59.530069 26508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41350
I20240501 19:59:59.530298 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:59:59.531661 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593576104309-24854-0/minicluster-data/ts-2-root/instance:
uuid: "42f869c3ee6e424093a58fc2ecd6268f"
format_stamp: "Formatted at 2024-05-01 19:59:59 on dist-test-slave-z8x3"
I20240501 19:59:59.535867 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240501 19:59:59.539021 26721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:59.539732 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:59:59.540004 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593576104309-24854-0/minicluster-data/ts-2-root
uuid: "42f869c3ee6e424093a58fc2ecd6268f"
format_stamp: "Formatted at 2024-05-01 19:59:59 on dist-test-slave-z8x3"
I20240501 19:59:59.540294 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593576104309-24854-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593576104309-24854-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593576104309-24854-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:59:59.549124 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:59:59.550061 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:59:59.551347 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:59:59.553444 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:59:59.553633 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:59.553850 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:59:59.553998 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:59:59.591650 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.131:44129
I20240501 19:59:59.591743 26783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.131:44129 every 8 connection(s)
I20240501 19:59:59.604748 26784 heartbeater.cc:347] Connected to a master server at 127.24.69.190:39931
I20240501 19:59:59.605147 26784 heartbeater.cc:464] Registering TS with master...
I20240501 19:59:59.605845 26784 heartbeater.cc:511] Master 127.24.69.190:39931 requested a full tablet report, sending...
I20240501 19:59:59.608213 26508 ts_manager.cc:194] Registered new tserver with Master: 42f869c3ee6e424093a58fc2ecd6268f (127.24.69.131:44129)
I20240501 19:59:59.608423 24854 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013842687s
I20240501 19:59:59.610020 26508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41352
I20240501 20:00:00.450352 26635 heartbeater.cc:503] Master 127.24.69.190:39931 was elected leader, sending a full tablet report...
I20240501 20:00:00.532744 26710 heartbeater.cc:503] Master 127.24.69.190:39931 was elected leader, sending a full tablet report...
I20240501 20:00:00.612574 26784 heartbeater.cc:503] Master 127.24.69.190:39931 was elected leader, sending a full tablet report...
I20240501 20:00:00.641275 24854 test_util.cc:268] Using random seed: -293594914
I20240501 20:00:00.675308 26508 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41358:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 20:00:00.677274 26508 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:00:00.720701 26600 tablet_service.cc:1467] Processing CreateTablet for tablet 1a6bd1d77970465487acd1abe91687ad (DEFAULT_TABLE table=test-workload [id=c386bd0ac6f64880b92a687c35ec7c5c]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240501 20:00:00.721088 26599 tablet_service.cc:1467] Processing CreateTablet for tablet 6bfaaa9fb25a4e18adf32f7339f33fa1 (DEFAULT_TABLE table=test-workload [id=c386bd0ac6f64880b92a687c35ec7c5c]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240501 20:00:00.722184 26600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a6bd1d77970465487acd1abe91687ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:00.722807 26599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bfaaa9fb25a4e18adf32f7339f33fa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:00.727339 26675 tablet_service.cc:1467] Processing CreateTablet for tablet 1a6bd1d77970465487acd1abe91687ad (DEFAULT_TABLE table=test-workload [id=c386bd0ac6f64880b92a687c35ec7c5c]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240501 20:00:00.728507 26675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a6bd1d77970465487acd1abe91687ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:00.729166 26674 tablet_service.cc:1467] Processing CreateTablet for tablet 6bfaaa9fb25a4e18adf32f7339f33fa1 (DEFAULT_TABLE table=test-workload [id=c386bd0ac6f64880b92a687c35ec7c5c]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240501 20:00:00.730394 26674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bfaaa9fb25a4e18adf32f7339f33fa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:00.731595 26749 tablet_service.cc:1467] Processing CreateTablet for tablet 1a6bd1d77970465487acd1abe91687ad (DEFAULT_TABLE table=test-workload [id=c386bd0ac6f64880b92a687c35ec7c5c]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240501 20:00:00.732874 26749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a6bd1d77970465487acd1abe91687ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:00.737323 26748 tablet_service.cc:1467] Processing CreateTablet for tablet 6bfaaa9fb25a4e18adf32f7339f33fa1 (DEFAULT_TABLE table=test-workload [id=c386bd0ac6f64880b92a687c35ec7c5c]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240501 20:00:00.738566 26748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bfaaa9fb25a4e18adf32f7339f33fa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:00.753146 26805 tablet_bootstrap.cc:492] T 1a6bd1d77970465487acd1abe91687ad P 42f869c3ee6e424093a58fc2ecd6268f: Bootstrap starting.
I20240501 20:00:00.758785 26805 tablet_bootstrap.cc:654] T 1a6bd1d77970465487acd1abe91687ad P 42f869c3ee6e424093a58fc2ecd6268f: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:00.775264 26806 tablet_bootstrap.cc:492] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df: Bootstrap starting.
I20240501 20:00:00.779793 26806 tablet_bootstrap.cc:654] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:00.780439 26804 tablet_bootstrap.cc:492] T 1a6bd1d77970465487acd1abe91687ad P 5f3e6e0ea51648bb90f831efd3bdd8de: Bootstrap starting.
I20240501 20:00:00.782569 26805 tablet_bootstrap.cc:492] T 1a6bd1d77970465487acd1abe91687ad P 42f869c3ee6e424093a58fc2ecd6268f: No bootstrap required, opened a new log
I20240501 20:00:00.783078 26805 ts_tablet_manager.cc:1367] T 1a6bd1d77970465487acd1abe91687ad P 42f869c3ee6e424093a58fc2ecd6268f: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.007s
I20240501 20:00:00.783946 26806 tablet_bootstrap.cc:492] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df: No bootstrap required, opened a new log
I20240501 20:00:00.784345 26806 ts_tablet_manager.cc:1367] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.000s
I20240501 20:00:00.785705 26804 tablet_bootstrap.cc:654] T 1a6bd1d77970465487acd1abe91687ad P 5f3e6e0ea51648bb90f831efd3bdd8de: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:00.786644 26806 raft_consensus.cc:348] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } }
I20240501 20:00:00.787269 26806 raft_consensus.cc:374] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:00.786659 26805 raft_consensus.cc:348] T 1a6bd1d77970465487acd1abe91687ad P 42f869c3ee6e424093a58fc2ecd6268f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } }
I20240501 20:00:00.788341 26806 raft_consensus.cc:729] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3a5fc4685a45e592318abadee052df, State: Initialized, Role: FOLLOWER
I20240501 20:00:00.788578 26805 raft_consensus.cc:374] T 1a6bd1d77970465487acd1abe91687ad P 42f869c3ee6e424093a58fc2ecd6268f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:00.789011 26805 raft_consensus.cc:729] T 1a6bd1d77970465487acd1abe91687ad P 42f869c3ee6e424093a58fc2ecd6268f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42f869c3ee6e424093a58fc2ecd6268f, State: Initialized, Role: FOLLOWER
I20240501 20:00:00.789018 26806 consensus_queue.cc:260] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } }
I20240501 20:00:00.789924 26805 consensus_queue.cc:260] T 1a6bd1d77970465487acd1abe91687ad P 42f869c3ee6e424093a58fc2ecd6268f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } }
I20240501 20:00:00.791334 26804 tablet_bootstrap.cc:492] T 1a6bd1d77970465487acd1abe91687ad P 5f3e6e0ea51648bb90f831efd3bdd8de: No bootstrap required, opened a new log
I20240501 20:00:00.791837 26804 ts_tablet_manager.cc:1367] T 1a6bd1d77970465487acd1abe91687ad P 5f3e6e0ea51648bb90f831efd3bdd8de: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.003s
I20240501 20:00:00.793045 26805 ts_tablet_manager.cc:1398] T 1a6bd1d77970465487acd1abe91687ad P 42f869c3ee6e424093a58fc2ecd6268f: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20240501 20:00:00.793253 26806 ts_tablet_manager.cc:1398] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240501 20:00:00.793946 26805 tablet_bootstrap.cc:492] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 42f869c3ee6e424093a58fc2ecd6268f: Bootstrap starting.
I20240501 20:00:00.794162 26806 tablet_bootstrap.cc:492] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df: Bootstrap starting.
I20240501 20:00:00.798445 26804 raft_consensus.cc:348] T 1a6bd1d77970465487acd1abe91687ad P 5f3e6e0ea51648bb90f831efd3bdd8de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } }
I20240501 20:00:00.799273 26804 raft_consensus.cc:374] T 1a6bd1d77970465487acd1abe91687ad P 5f3e6e0ea51648bb90f831efd3bdd8de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:00.799634 26804 raft_consensus.cc:729] T 1a6bd1d77970465487acd1abe91687ad P 5f3e6e0ea51648bb90f831efd3bdd8de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f3e6e0ea51648bb90f831efd3bdd8de, State: Initialized, Role: FOLLOWER
I20240501 20:00:00.799763 26805 tablet_bootstrap.cc:654] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 42f869c3ee6e424093a58fc2ecd6268f: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:00.800601 26804 consensus_queue.cc:260] T 1a6bd1d77970465487acd1abe91687ad P 5f3e6e0ea51648bb90f831efd3bdd8de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } }
I20240501 20:00:00.802544 26806 tablet_bootstrap.cc:654] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:00.810587 26804 ts_tablet_manager.cc:1398] T 1a6bd1d77970465487acd1abe91687ad P 5f3e6e0ea51648bb90f831efd3bdd8de: Time spent starting tablet: real 0.018s user 0.001s sys 0.011s
I20240501 20:00:00.813851 26805 tablet_bootstrap.cc:492] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 42f869c3ee6e424093a58fc2ecd6268f: No bootstrap required, opened a new log
I20240501 20:00:00.814323 26805 ts_tablet_manager.cc:1367] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 42f869c3ee6e424093a58fc2ecd6268f: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.004s
I20240501 20:00:00.815929 26804 tablet_bootstrap.cc:492] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 5f3e6e0ea51648bb90f831efd3bdd8de: Bootstrap starting.
I20240501 20:00:00.816810 26805 raft_consensus.cc:348] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 42f869c3ee6e424093a58fc2ecd6268f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } }
I20240501 20:00:00.817565 26805 raft_consensus.cc:374] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 42f869c3ee6e424093a58fc2ecd6268f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:00.817858 26805 raft_consensus.cc:729] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 42f869c3ee6e424093a58fc2ecd6268f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42f869c3ee6e424093a58fc2ecd6268f, State: Initialized, Role: FOLLOWER
I20240501 20:00:00.818552 26805 consensus_queue.cc:260] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 42f869c3ee6e424093a58fc2ecd6268f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } }
I20240501 20:00:00.820652 26805 ts_tablet_manager.cc:1398] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 42f869c3ee6e424093a58fc2ecd6268f: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20240501 20:00:00.823434 26806 tablet_bootstrap.cc:492] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df: No bootstrap required, opened a new log
I20240501 20:00:00.823617 26804 tablet_bootstrap.cc:654] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 5f3e6e0ea51648bb90f831efd3bdd8de: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:00.824046 26806 ts_tablet_manager.cc:1367] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.016s
I20240501 20:00:00.827004 26806 raft_consensus.cc:348] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } }
I20240501 20:00:00.828061 26806 raft_consensus.cc:374] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:00.828502 26806 raft_consensus.cc:729] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3a5fc4685a45e592318abadee052df, State: Initialized, Role: FOLLOWER
I20240501 20:00:00.829227 26806 consensus_queue.cc:260] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } }
I20240501 20:00:00.830888 26804 tablet_bootstrap.cc:492] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 5f3e6e0ea51648bb90f831efd3bdd8de: No bootstrap required, opened a new log
I20240501 20:00:00.831245 26806 ts_tablet_manager.cc:1398] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240501 20:00:00.831354 26804 ts_tablet_manager.cc:1367] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 5f3e6e0ea51648bb90f831efd3bdd8de: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20240501 20:00:00.833256 26804 raft_consensus.cc:348] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 5f3e6e0ea51648bb90f831efd3bdd8de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } }
I20240501 20:00:00.833740 26804 raft_consensus.cc:374] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 5f3e6e0ea51648bb90f831efd3bdd8de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:00.833973 26804 raft_consensus.cc:729] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 5f3e6e0ea51648bb90f831efd3bdd8de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f3e6e0ea51648bb90f831efd3bdd8de, State: Initialized, Role: FOLLOWER
I20240501 20:00:00.834434 26804 consensus_queue.cc:260] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 5f3e6e0ea51648bb90f831efd3bdd8de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } }
I20240501 20:00:00.836099 26804 ts_tablet_manager.cc:1398] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 5f3e6e0ea51648bb90f831efd3bdd8de: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:00:00.868980 26810 raft_consensus.cc:482] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:00.869390 26810 raft_consensus.cc:504] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } }
I20240501 20:00:00.871315 26810 leader_election.cc:288] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f3e6e0ea51648bb90f831efd3bdd8de (127.24.69.129:34497), 42f869c3ee6e424093a58fc2ecd6268f (127.24.69.131:44129)
I20240501 20:00:00.875661 26810 raft_consensus.cc:482] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:00.876122 26810 raft_consensus.cc:504] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } }
I20240501 20:00:00.877915 26810 leader_election.cc:288] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f3e6e0ea51648bb90f831efd3bdd8de (127.24.69.129:34497), 42f869c3ee6e424093a58fc2ecd6268f (127.24.69.131:44129)
I20240501 20:00:00.881650 26610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a6bd1d77970465487acd1abe91687ad" candidate_uuid: "ab3a5fc4685a45e592318abadee052df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" is_pre_election: true
I20240501 20:00:00.881915 26609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bfaaa9fb25a4e18adf32f7339f33fa1" candidate_uuid: "ab3a5fc4685a45e592318abadee052df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" is_pre_election: true
I20240501 20:00:00.882261 26610 raft_consensus.cc:2454] T 1a6bd1d77970465487acd1abe91687ad P 5f3e6e0ea51648bb90f831efd3bdd8de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3a5fc4685a45e592318abadee052df in term 0.
I20240501 20:00:00.882820 26609 raft_consensus.cc:2454] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 5f3e6e0ea51648bb90f831efd3bdd8de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3a5fc4685a45e592318abadee052df in term 0.
I20240501 20:00:00.883522 26759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a6bd1d77970465487acd1abe91687ad" candidate_uuid: "ab3a5fc4685a45e592318abadee052df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42f869c3ee6e424093a58fc2ecd6268f" is_pre_election: true
I20240501 20:00:00.883919 26758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bfaaa9fb25a4e18adf32f7339f33fa1" candidate_uuid: "ab3a5fc4685a45e592318abadee052df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42f869c3ee6e424093a58fc2ecd6268f" is_pre_election: true
I20240501 20:00:00.884236 26759 raft_consensus.cc:2454] T 1a6bd1d77970465487acd1abe91687ad P 42f869c3ee6e424093a58fc2ecd6268f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3a5fc4685a45e592318abadee052df in term 0.
I20240501 20:00:00.884665 26758 raft_consensus.cc:2454] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 42f869c3ee6e424093a58fc2ecd6268f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3a5fc4685a45e592318abadee052df in term 0.
I20240501 20:00:00.884670 26651 leader_election.cc:302] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f3e6e0ea51648bb90f831efd3bdd8de, ab3a5fc4685a45e592318abadee052df; no voters:
I20240501 20:00:00.885702 26810 raft_consensus.cc:2781] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:00.885828 26651 leader_election.cc:302] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f3e6e0ea51648bb90f831efd3bdd8de, ab3a5fc4685a45e592318abadee052df; no voters:
I20240501 20:00:00.886080 26810 raft_consensus.cc:482] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:00.886426 26810 raft_consensus.cc:3037] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:00.887472 26818 raft_consensus.cc:2781] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:00.887782 26818 raft_consensus.cc:482] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:00.888033 26818 raft_consensus.cc:3037] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:00.891309 26810 raft_consensus.cc:504] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } }
I20240501 20:00:00.892045 26818 raft_consensus.cc:504] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } }
I20240501 20:00:00.892802 26810 leader_election.cc:288] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [CANDIDATE]: Term 1 election: Requested vote from peers 5f3e6e0ea51648bb90f831efd3bdd8de (127.24.69.129:34497), 42f869c3ee6e424093a58fc2ecd6268f (127.24.69.131:44129)
I20240501 20:00:00.893568 26609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a6bd1d77970465487acd1abe91687ad" candidate_uuid: "ab3a5fc4685a45e592318abadee052df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de"
I20240501 20:00:00.893718 26818 leader_election.cc:288] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [CANDIDATE]: Term 1 election: Requested vote from peers 5f3e6e0ea51648bb90f831efd3bdd8de (127.24.69.129:34497), 42f869c3ee6e424093a58fc2ecd6268f (127.24.69.131:44129)
I20240501 20:00:00.893738 26758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a6bd1d77970465487acd1abe91687ad" candidate_uuid: "ab3a5fc4685a45e592318abadee052df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42f869c3ee6e424093a58fc2ecd6268f"
I20240501 20:00:00.894493 26758 raft_consensus.cc:3037] T 1a6bd1d77970465487acd1abe91687ad P 42f869c3ee6e424093a58fc2ecd6268f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:00.894662 26609 raft_consensus.cc:3037] T 1a6bd1d77970465487acd1abe91687ad P 5f3e6e0ea51648bb90f831efd3bdd8de [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:00.894696 26759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bfaaa9fb25a4e18adf32f7339f33fa1" candidate_uuid: "ab3a5fc4685a45e592318abadee052df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42f869c3ee6e424093a58fc2ecd6268f"
I20240501 20:00:00.894817 26610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bfaaa9fb25a4e18adf32f7339f33fa1" candidate_uuid: "ab3a5fc4685a45e592318abadee052df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de"
I20240501 20:00:00.895452 26759 raft_consensus.cc:3037] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 42f869c3ee6e424093a58fc2ecd6268f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:00.895623 26610 raft_consensus.cc:3037] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 5f3e6e0ea51648bb90f831efd3bdd8de [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:00.900959 26758 raft_consensus.cc:2454] T 1a6bd1d77970465487acd1abe91687ad P 42f869c3ee6e424093a58fc2ecd6268f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3a5fc4685a45e592318abadee052df in term 1.
I20240501 20:00:00.901480 26609 raft_consensus.cc:2454] T 1a6bd1d77970465487acd1abe91687ad P 5f3e6e0ea51648bb90f831efd3bdd8de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3a5fc4685a45e592318abadee052df in term 1.
I20240501 20:00:00.901786 26759 raft_consensus.cc:2454] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 42f869c3ee6e424093a58fc2ecd6268f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3a5fc4685a45e592318abadee052df in term 1.
I20240501 20:00:00.902146 26651 leader_election.cc:302] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42f869c3ee6e424093a58fc2ecd6268f, ab3a5fc4685a45e592318abadee052df; no voters:
I20240501 20:00:00.902278 26610 raft_consensus.cc:2454] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 5f3e6e0ea51648bb90f831efd3bdd8de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3a5fc4685a45e592318abadee052df in term 1.
I20240501 20:00:00.902920 26818 raft_consensus.cc:2781] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:00.903359 26818 raft_consensus.cc:686] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [term 1 LEADER]: Becoming Leader. State: Replica: ab3a5fc4685a45e592318abadee052df, State: Running, Role: LEADER
I20240501 20:00:00.903365 26651 leader_election.cc:302] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42f869c3ee6e424093a58fc2ecd6268f, ab3a5fc4685a45e592318abadee052df; no voters:
I20240501 20:00:00.904284 26810 raft_consensus.cc:2781] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:00.904299 26818 consensus_queue.cc:237] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } }
I20240501 20:00:00.905616 26810 raft_consensus.cc:686] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [term 1 LEADER]: Becoming Leader. State: Replica: ab3a5fc4685a45e592318abadee052df, State: Running, Role: LEADER
I20240501 20:00:00.906406 26810 consensus_queue.cc:237] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } }
I20240501 20:00:00.913769 26506 catalog_manager.cc:5484] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df reported cstate change: term changed from 0 to 1, leader changed from <none> to ab3a5fc4685a45e592318abadee052df (127.24.69.130). New cstate: current_term: 1 leader_uuid: "ab3a5fc4685a45e592318abadee052df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:00.923892 26505 catalog_manager.cc:5484] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df reported cstate change: term changed from 0 to 1, leader changed from <none> to ab3a5fc4685a45e592318abadee052df (127.24.69.130). New cstate: current_term: 1 leader_uuid: "ab3a5fc4685a45e592318abadee052df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:01.249552 26561 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:01.260706 26685 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6bfaaa9fb25a4e18adf32f7339f33fa1"
dest_uuid: "ab3a5fc4685a45e592318abadee052df"
mode: GRACEFUL
new_leader_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de"
from {username='slave'} at 127.0.0.1:35534
I20240501 20:00:01.261234 26685 raft_consensus.cc:595] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [term 1 LEADER]: Received request to transfer leadership to TS 5f3e6e0ea51648bb90f831efd3bdd8de
I20240501 20:00:01.262221 26561 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 20:00:01.262529 26561 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:01.353089 26810 consensus_queue.cc:1035] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:01.359190 26818 raft_consensus.cc:979] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df: : Instructing follower 5f3e6e0ea51648bb90f831efd3bdd8de to start an election
I20240501 20:00:01.359645 26819 raft_consensus.cc:1067] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [term 1 LEADER]: Signalling peer 5f3e6e0ea51648bb90f831efd3bdd8de to start an election
I20240501 20:00:01.362095 26609 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6bfaaa9fb25a4e18adf32f7339f33fa1"
dest_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de"
from {username='slave'} at 127.0.0.1:34140
I20240501 20:00:01.363368 26609 raft_consensus.cc:482] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 5f3e6e0ea51648bb90f831efd3bdd8de [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:01.363770 26609 raft_consensus.cc:3037] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 5f3e6e0ea51648bb90f831efd3bdd8de [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:01.365423 26819 consensus_queue.cc:1035] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [LEADER]: Connected to new peer: Peer: permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:01.370270 26609 raft_consensus.cc:504] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 5f3e6e0ea51648bb90f831efd3bdd8de [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } }
I20240501 20:00:01.372571 26609 leader_election.cc:288] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 5f3e6e0ea51648bb90f831efd3bdd8de [CANDIDATE]: Term 2 election: Requested vote from peers ab3a5fc4685a45e592318abadee052df (127.24.69.130:33453), 42f869c3ee6e424093a58fc2ecd6268f (127.24.69.131:44129)
I20240501 20:00:01.383620 26819 consensus_queue.cc:1035] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [LEADER]: Connected to new peer: Peer: permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:01.386603 26685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bfaaa9fb25a4e18adf32f7339f33fa1" candidate_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab3a5fc4685a45e592318abadee052df"
I20240501 20:00:01.387195 26685 raft_consensus.cc:3032] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:00:01.387513 26685 raft_consensus.cc:729] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab3a5fc4685a45e592318abadee052df, State: Running, Role: LEADER
I20240501 20:00:01.388290 26685 consensus_queue.cc:260] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } }
I20240501 20:00:01.389708 26685 raft_consensus.cc:3037] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:01.392611 26759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bfaaa9fb25a4e18adf32f7339f33fa1" candidate_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "42f869c3ee6e424093a58fc2ecd6268f"
I20240501 20:00:01.393193 26759 raft_consensus.cc:3037] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 42f869c3ee6e424093a58fc2ecd6268f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:01.396034 26685 raft_consensus.cc:2454] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f3e6e0ea51648bb90f831efd3bdd8de in term 2.
I20240501 20:00:01.397469 26575 leader_election.cc:302] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 5f3e6e0ea51648bb90f831efd3bdd8de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f3e6e0ea51648bb90f831efd3bdd8de, ab3a5fc4685a45e592318abadee052df; no voters:
I20240501 20:00:01.398269 26824 raft_consensus.cc:2781] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 5f3e6e0ea51648bb90f831efd3bdd8de [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:00:01.399262 26759 raft_consensus.cc:2454] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 42f869c3ee6e424093a58fc2ecd6268f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f3e6e0ea51648bb90f831efd3bdd8de in term 2.
I20240501 20:00:01.399855 26818 consensus_queue.cc:1035] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:01.400959 26824 raft_consensus.cc:686] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 5f3e6e0ea51648bb90f831efd3bdd8de [term 2 LEADER]: Becoming Leader. State: Replica: 5f3e6e0ea51648bb90f831efd3bdd8de, State: Running, Role: LEADER
I20240501 20:00:01.401715 26824 consensus_queue.cc:237] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 5f3e6e0ea51648bb90f831efd3bdd8de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } }
I20240501 20:00:01.409662 26505 catalog_manager.cc:5484] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 5f3e6e0ea51648bb90f831efd3bdd8de reported cstate change: term changed from 1 to 2, leader changed from ab3a5fc4685a45e592318abadee052df (127.24.69.130) to 5f3e6e0ea51648bb90f831efd3bdd8de (127.24.69.129). New cstate: current_term: 2 leader_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:01.797955 26685 raft_consensus.cc:1261] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [term 2 FOLLOWER]: Refusing update from remote peer 5f3e6e0ea51648bb90f831efd3bdd8de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:01.799178 26824 consensus_queue.cc:1035] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 5f3e6e0ea51648bb90f831efd3bdd8de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:01.807096 26759 raft_consensus.cc:1261] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 42f869c3ee6e424093a58fc2ecd6268f [term 2 FOLLOWER]: Refusing update from remote peer 5f3e6e0ea51648bb90f831efd3bdd8de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:01.808995 26824 consensus_queue.cc:1035] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 5f3e6e0ea51648bb90f831efd3bdd8de [LEADER]: Connected to new peer: Peer: permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:03.263258 26561 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:03.265199 26561 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:03.265512 26561 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:05.266243 26561 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:05.268035 26561 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:05.268388 26561 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:07.269414 26561 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:07.272558 26561 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:07.272989 26561 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:09.273950 26561 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:09.275645 26561 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:09.275964 26561 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:11.276855 26561 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:11.278610 26561 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:11.278892 26561 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:12.032577 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:12.040184 26842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:12.041482 26843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:12.042469 26845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:12.043498 24854 server_base.cc:1034] running on GCE node
I20240501 20:00:12.044505 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:00:12.044749 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:00:12.044920 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593612044905 us; error 0 us; skew 500 ppm
I20240501 20:00:12.045478 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:00:12.053800 24854 webserver.cc:460] Webserver started at http://127.24.69.132:33021/ using document root <none> and password file <none>
I20240501 20:00:12.054294 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:12.054486 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:12.054859 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:12.056310 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593576104309-24854-0/minicluster-data/ts-3-root/instance:
uuid: "48fea2b9590b473c87f4db0bf06a88bf"
format_stamp: "Formatted at 2024-05-01 20:00:12 on dist-test-slave-z8x3"
I20240501 20:00:12.060838 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240501 20:00:12.064093 26850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:12.064860 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 20:00:12.065136 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593576104309-24854-0/minicluster-data/ts-3-root
uuid: "48fea2b9590b473c87f4db0bf06a88bf"
format_stamp: "Formatted at 2024-05-01 20:00:12 on dist-test-slave-z8x3"
I20240501 20:00:12.065405 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593576104309-24854-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593576104309-24854-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593576104309-24854-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:12.079313 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:12.080689 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:12.082830 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:12.085283 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:12.085492 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:12.085726 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:12.085892 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:12.124017 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.132:44337
I20240501 20:00:12.124115 26912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.132:44337 every 8 connection(s)
I20240501 20:00:12.139174 26913 heartbeater.cc:347] Connected to a master server at 127.24.69.190:39931
I20240501 20:00:12.139528 26913 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:12.140195 26913 heartbeater.cc:511] Master 127.24.69.190:39931 requested a full tablet report, sending...
I20240501 20:00:12.142156 26505 ts_manager.cc:194] Registered new tserver with Master: 48fea2b9590b473c87f4db0bf06a88bf (127.24.69.132:44337)
I20240501 20:00:12.143738 26505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55690
I20240501 20:00:12.308863 26685 consensus_queue.cc:237] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } attrs { replace: true } } peers { permanent_uuid: "48fea2b9590b473c87f4db0bf06a88bf" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 44337 } attrs { promote: true } }
I20240501 20:00:12.317215 26759 raft_consensus.cc:1261] T 1a6bd1d77970465487acd1abe91687ad P 42f869c3ee6e424093a58fc2ecd6268f [term 1 FOLLOWER]: Refusing update from remote peer ab3a5fc4685a45e592318abadee052df: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:00:12.319569 26840 consensus_queue.cc:1035] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [LEADER]: Connected to new peer: Peer: permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:00:12.330258 26609 raft_consensus.cc:1261] T 1a6bd1d77970465487acd1abe91687ad P 5f3e6e0ea51648bb90f831efd3bdd8de [term 1 FOLLOWER]: Refusing update from remote peer ab3a5fc4685a45e592318abadee052df: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:00:12.332458 26840 consensus_queue.cc:1035] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:12.332187 26920 raft_consensus.cc:2932] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 48fea2b9590b473c87f4db0bf06a88bf (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } attrs { replace: true } } peers { permanent_uuid: "48fea2b9590b473c87f4db0bf06a88bf" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 44337 } attrs { promote: true } } }
I20240501 20:00:12.334184 26759 raft_consensus.cc:2932] T 1a6bd1d77970465487acd1abe91687ad P 42f869c3ee6e424093a58fc2ecd6268f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 48fea2b9590b473c87f4db0bf06a88bf (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } attrs { replace: true } } peers { permanent_uuid: "48fea2b9590b473c87f4db0bf06a88bf" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 44337 } attrs { promote: true } } }
W20240501 20:00:12.343966 26648 consensus_peers.cc:485] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df -> Peer 48fea2b9590b473c87f4db0bf06a88bf (127.24.69.132:44337): Couldn't send request to peer 48fea2b9590b473c87f4db0bf06a88bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a6bd1d77970465487acd1abe91687ad. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:00:12.345856 26609 raft_consensus.cc:2932] T 1a6bd1d77970465487acd1abe91687ad P 5f3e6e0ea51648bb90f831efd3bdd8de [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 48fea2b9590b473c87f4db0bf06a88bf (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } attrs { replace: true } } peers { permanent_uuid: "48fea2b9590b473c87f4db0bf06a88bf" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 44337 } attrs { promote: true } } }
I20240501 20:00:12.349773 26505 catalog_manager.cc:5484] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df reported cstate change: config changed from index -1 to 2, NON_VOTER 48fea2b9590b473c87f4db0bf06a88bf (127.24.69.132) added. New cstate: current_term: 1 leader_uuid: "ab3a5fc4685a45e592318abadee052df" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48fea2b9590b473c87f4db0bf06a88bf" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 44337 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:12.731611 26931 ts_tablet_manager.cc:902] T 1a6bd1d77970465487acd1abe91687ad P 48fea2b9590b473c87f4db0bf06a88bf: Initiating tablet copy from peer ab3a5fc4685a45e592318abadee052df (127.24.69.130:33453)
I20240501 20:00:12.735415 26931 tablet_copy_client.cc:308] T 1a6bd1d77970465487acd1abe91687ad P 48fea2b9590b473c87f4db0bf06a88bf: tablet copy: Beginning tablet copy session from remote peer at address 127.24.69.130:33453
I20240501 20:00:12.747896 26695 tablet_copy_service.cc:140] P ab3a5fc4685a45e592318abadee052df: Received BeginTabletCopySession request for tablet 1a6bd1d77970465487acd1abe91687ad from peer 48fea2b9590b473c87f4db0bf06a88bf ({username='slave'} at 127.0.0.1:36800)
I20240501 20:00:12.748554 26695 tablet_copy_service.cc:161] P ab3a5fc4685a45e592318abadee052df: Beginning new tablet copy session on tablet 1a6bd1d77970465487acd1abe91687ad from peer 48fea2b9590b473c87f4db0bf06a88bf at {username='slave'} at 127.0.0.1:36800: session id = 48fea2b9590b473c87f4db0bf06a88bf-1a6bd1d77970465487acd1abe91687ad
I20240501 20:00:12.755200 26695 tablet_copy_source_session.cc:205] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:00:12.758195 26931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a6bd1d77970465487acd1abe91687ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:12.770119 26931 tablet_copy_client.cc:783] T 1a6bd1d77970465487acd1abe91687ad P 48fea2b9590b473c87f4db0bf06a88bf: tablet copy: Starting download of 0 data blocks...
I20240501 20:00:12.770664 26931 tablet_copy_client.cc:647] T 1a6bd1d77970465487acd1abe91687ad P 48fea2b9590b473c87f4db0bf06a88bf: tablet copy: Starting download of 1 WAL segments...
I20240501 20:00:12.774149 26931 tablet_copy_client.cc:523] T 1a6bd1d77970465487acd1abe91687ad P 48fea2b9590b473c87f4db0bf06a88bf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:00:12.780614 26931 tablet_bootstrap.cc:492] T 1a6bd1d77970465487acd1abe91687ad P 48fea2b9590b473c87f4db0bf06a88bf: Bootstrap starting.
I20240501 20:00:12.797645 26931 tablet_bootstrap.cc:492] T 1a6bd1d77970465487acd1abe91687ad P 48fea2b9590b473c87f4db0bf06a88bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:00:12.798416 26931 tablet_bootstrap.cc:492] T 1a6bd1d77970465487acd1abe91687ad P 48fea2b9590b473c87f4db0bf06a88bf: Bootstrap complete.
I20240501 20:00:12.799006 26931 ts_tablet_manager.cc:1367] T 1a6bd1d77970465487acd1abe91687ad P 48fea2b9590b473c87f4db0bf06a88bf: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.000s
I20240501 20:00:12.801247 26931 raft_consensus.cc:348] T 1a6bd1d77970465487acd1abe91687ad P 48fea2b9590b473c87f4db0bf06a88bf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } attrs { replace: true } } peers { permanent_uuid: "48fea2b9590b473c87f4db0bf06a88bf" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 44337 } attrs { promote: true } }
I20240501 20:00:12.801890 26931 raft_consensus.cc:729] T 1a6bd1d77970465487acd1abe91687ad P 48fea2b9590b473c87f4db0bf06a88bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 48fea2b9590b473c87f4db0bf06a88bf, State: Initialized, Role: LEARNER
I20240501 20:00:12.802454 26931 consensus_queue.cc:260] T 1a6bd1d77970465487acd1abe91687ad P 48fea2b9590b473c87f4db0bf06a88bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } attrs { replace: true } } peers { permanent_uuid: "48fea2b9590b473c87f4db0bf06a88bf" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 44337 } attrs { promote: true } }
I20240501 20:00:12.804720 26913 heartbeater.cc:503] Master 127.24.69.190:39931 was elected leader, sending a full tablet report...
I20240501 20:00:12.805130 26931 ts_tablet_manager.cc:1398] T 1a6bd1d77970465487acd1abe91687ad P 48fea2b9590b473c87f4db0bf06a88bf: Time spent starting tablet: real 0.006s user 0.003s sys 0.005s
I20240501 20:00:12.807173 26695 tablet_copy_service.cc:342] P ab3a5fc4685a45e592318abadee052df: Request end of tablet copy session 48fea2b9590b473c87f4db0bf06a88bf-1a6bd1d77970465487acd1abe91687ad received from {username='slave'} at 127.0.0.1:36800
I20240501 20:00:12.807507 26695 tablet_copy_service.cc:433] P ab3a5fc4685a45e592318abadee052df: ending tablet copy session 48fea2b9590b473c87f4db0bf06a88bf-1a6bd1d77970465487acd1abe91687ad on tablet 1a6bd1d77970465487acd1abe91687ad with peer 48fea2b9590b473c87f4db0bf06a88bf
I20240501 20:00:13.279557 26561 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:13.281415 26561 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:13.281765 26561 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:13.294018 26888 raft_consensus.cc:1203] T 1a6bd1d77970465487acd1abe91687ad P 48fea2b9590b473c87f4db0bf06a88bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:00:13.833439 26940 raft_consensus.cc:1050] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df: attempting to promote NON_VOTER 48fea2b9590b473c87f4db0bf06a88bf to VOTER
I20240501 20:00:13.835351 26940 consensus_queue.cc:237] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } attrs { replace: true } } peers { permanent_uuid: "48fea2b9590b473c87f4db0bf06a88bf" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 44337 } attrs { promote: false } }
I20240501 20:00:13.841004 26888 raft_consensus.cc:1261] T 1a6bd1d77970465487acd1abe91687ad P 48fea2b9590b473c87f4db0bf06a88bf [term 1 LEARNER]: Refusing update from remote peer ab3a5fc4685a45e592318abadee052df: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:00:13.841936 26759 raft_consensus.cc:1261] T 1a6bd1d77970465487acd1abe91687ad P 42f869c3ee6e424093a58fc2ecd6268f [term 1 FOLLOWER]: Refusing update from remote peer ab3a5fc4685a45e592318abadee052df: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:00:13.842761 26941 consensus_queue.cc:1035] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [LEADER]: Connected to new peer: Peer: permanent_uuid: "48fea2b9590b473c87f4db0bf06a88bf" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:00:13.845101 26943 consensus_queue.cc:1035] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [LEADER]: Connected to new peer: Peer: permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:00:13.849539 26609 raft_consensus.cc:1261] T 1a6bd1d77970465487acd1abe91687ad P 5f3e6e0ea51648bb90f831efd3bdd8de [term 1 FOLLOWER]: Refusing update from remote peer ab3a5fc4685a45e592318abadee052df: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:00:13.850878 26941 consensus_queue.cc:1035] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:00:13.860029 26944 raft_consensus.cc:2932] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 48fea2b9590b473c87f4db0bf06a88bf (127.24.69.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } attrs { replace: true } } peers { permanent_uuid: "48fea2b9590b473c87f4db0bf06a88bf" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 44337 } attrs { promote: false } } }
I20240501 20:00:13.862279 26888 raft_consensus.cc:2932] T 1a6bd1d77970465487acd1abe91687ad P 48fea2b9590b473c87f4db0bf06a88bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 48fea2b9590b473c87f4db0bf06a88bf (127.24.69.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } attrs { replace: true } } peers { permanent_uuid: "48fea2b9590b473c87f4db0bf06a88bf" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 44337 } attrs { promote: false } } }
I20240501 20:00:13.863612 26610 raft_consensus.cc:2932] T 1a6bd1d77970465487acd1abe91687ad P 5f3e6e0ea51648bb90f831efd3bdd8de [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 48fea2b9590b473c87f4db0bf06a88bf (127.24.69.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } attrs { replace: true } } peers { permanent_uuid: "48fea2b9590b473c87f4db0bf06a88bf" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 44337 } attrs { promote: false } } }
I20240501 20:00:13.873379 26506 catalog_manager.cc:5484] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df reported cstate change: config changed from index 2 to 3, 48fea2b9590b473c87f4db0bf06a88bf (127.24.69.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3a5fc4685a45e592318abadee052df" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48fea2b9590b473c87f4db0bf06a88bf" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 44337 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:00:13.872682 26759 raft_consensus.cc:2932] T 1a6bd1d77970465487acd1abe91687ad P 42f869c3ee6e424093a58fc2ecd6268f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 48fea2b9590b473c87f4db0bf06a88bf (127.24.69.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "42f869c3ee6e424093a58fc2ecd6268f" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 44129 } attrs { replace: true } } peers { permanent_uuid: "48fea2b9590b473c87f4db0bf06a88bf" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 44337 } attrs { promote: false } } }
I20240501 20:00:13.894665 26684 consensus_queue.cc:237] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "48fea2b9590b473c87f4db0bf06a88bf" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 44337 } attrs { promote: false } }
I20240501 20:00:13.899441 26888 raft_consensus.cc:1261] T 1a6bd1d77970465487acd1abe91687ad P 48fea2b9590b473c87f4db0bf06a88bf [term 1 FOLLOWER]: Refusing update from remote peer ab3a5fc4685a45e592318abadee052df: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:00:13.899601 26610 raft_consensus.cc:1261] T 1a6bd1d77970465487acd1abe91687ad P 5f3e6e0ea51648bb90f831efd3bdd8de [term 1 FOLLOWER]: Refusing update from remote peer ab3a5fc4685a45e592318abadee052df: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:00:13.901343 26944 consensus_queue.cc:1035] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:00:13.903105 26944 consensus_queue.cc:1035] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [LEADER]: Connected to new peer: Peer: permanent_uuid: "48fea2b9590b473c87f4db0bf06a88bf" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:00:13.908982 26941 raft_consensus.cc:2932] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 42f869c3ee6e424093a58fc2ecd6268f (127.24.69.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "48fea2b9590b473c87f4db0bf06a88bf" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 44337 } attrs { promote: false } } }
I20240501 20:00:13.910647 26888 raft_consensus.cc:2932] T 1a6bd1d77970465487acd1abe91687ad P 48fea2b9590b473c87f4db0bf06a88bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 42f869c3ee6e424093a58fc2ecd6268f (127.24.69.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "48fea2b9590b473c87f4db0bf06a88bf" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 44337 } attrs { promote: false } } }
I20240501 20:00:13.917805 26610 raft_consensus.cc:2932] T 1a6bd1d77970465487acd1abe91687ad P 5f3e6e0ea51648bb90f831efd3bdd8de [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 42f869c3ee6e424093a58fc2ecd6268f (127.24.69.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } } peers { permanent_uuid: "48fea2b9590b473c87f4db0bf06a88bf" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 44337 } attrs { promote: false } } }
I20240501 20:00:13.919595 26494 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1a6bd1d77970465487acd1abe91687ad with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:00:13.921514 26505 catalog_manager.cc:5484] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df reported cstate change: config changed from index 3 to 4, VOTER 42f869c3ee6e424093a58fc2ecd6268f (127.24.69.131) evicted. New cstate: current_term: 1 leader_uuid: "ab3a5fc4685a45e592318abadee052df" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3a5fc4685a45e592318abadee052df" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 33453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f3e6e0ea51648bb90f831efd3bdd8de" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 34497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48fea2b9590b473c87f4db0bf06a88bf" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 44337 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:00:13.931550 26748 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a6bd1d77970465487acd1abe91687ad with delete_type TABLET_DATA_TOMBSTONED (TS 42f869c3ee6e424093a58fc2ecd6268f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59324
I20240501 20:00:13.934356 26954 tablet_replica.cc:330] T 1a6bd1d77970465487acd1abe91687ad P 42f869c3ee6e424093a58fc2ecd6268f: stopping tablet replica
I20240501 20:00:13.935264 26954 raft_consensus.cc:2229] T 1a6bd1d77970465487acd1abe91687ad P 42f869c3ee6e424093a58fc2ecd6268f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:13.935876 26954 raft_consensus.cc:2258] T 1a6bd1d77970465487acd1abe91687ad P 42f869c3ee6e424093a58fc2ecd6268f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:13.938555 26954 ts_tablet_manager.cc:1875] T 1a6bd1d77970465487acd1abe91687ad P 42f869c3ee6e424093a58fc2ecd6268f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:00:13.948407 26954 ts_tablet_manager.cc:1888] T 1a6bd1d77970465487acd1abe91687ad P 42f869c3ee6e424093a58fc2ecd6268f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:00:13.948679 26954 log.cc:1196] T 1a6bd1d77970465487acd1abe91687ad P 42f869c3ee6e424093a58fc2ecd6268f: Deleting WAL directory at /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593576104309-24854-0/minicluster-data/ts-2-root/wals/1a6bd1d77970465487acd1abe91687ad
I20240501 20:00:13.949940 26495 catalog_manager.cc:4830] TS 42f869c3ee6e424093a58fc2ecd6268f (127.24.69.131:44129): tablet 1a6bd1d77970465487acd1abe91687ad (table test-workload [id=c386bd0ac6f64880b92a687c35ec7c5c]) successfully deleted
I20240501 20:00:15.282665 26561 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:15.284611 26561 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:15.284914 26561 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:17.285929 26561 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:17.288471 26561 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:17.288806 26561 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:19.289657 26561 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:19.291553 26561 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:19.291868 26561 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:21.623430 26561 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:21.626062 26561 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:21.626538 26561 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240501 20:00:22.305708 26706 debug-util.cc:398] Leaking SignalData structure 0x7b080009e940 after lost signal to thread 24857
W20240501 20:00:22.306614 26706 debug-util.cc:398] Leaking SignalData structure 0x7b080009d000 after lost signal to thread 26542
W20240501 20:00:22.307533 26706 debug-util.cc:398] Leaking SignalData structure 0x7b08001ff320 after lost signal to thread 26634
W20240501 20:00:22.308408 26706 debug-util.cc:398] Leaking SignalData structure 0x7b08000e7440 after lost signal to thread 26709
W20240501 20:00:22.309188 26706 debug-util.cc:398] Leaking SignalData structure 0x7b0800168400 after lost signal to thread 26783
W20240501 20:00:22.309993 26706 debug-util.cc:398] Leaking SignalData structure 0x7b080023ba20 after lost signal to thread 26912
I20240501 20:00:23.627641 26561 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:23.630628 26561 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:23.631054 26561 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:25.631932 26561 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:25.633813 26561 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:25.634107 26561 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:26.462733 24854 tablet_server.cc:178] TabletServer@127.24.69.129:0 shutting down...
I20240501 20:00:26.498220 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:00:26.499158 24854 tablet_replica.cc:330] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 5f3e6e0ea51648bb90f831efd3bdd8de: stopping tablet replica
I20240501 20:00:26.500006 24854 raft_consensus.cc:2229] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 5f3e6e0ea51648bb90f831efd3bdd8de [term 2 LEADER]: Raft consensus shutting down.
I20240501 20:00:26.501344 24854 raft_consensus.cc:2258] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 5f3e6e0ea51648bb90f831efd3bdd8de [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:26.503849 24854 tablet_replica.cc:330] T 1a6bd1d77970465487acd1abe91687ad P 5f3e6e0ea51648bb90f831efd3bdd8de: stopping tablet replica
I20240501 20:00:26.504344 24854 raft_consensus.cc:2229] T 1a6bd1d77970465487acd1abe91687ad P 5f3e6e0ea51648bb90f831efd3bdd8de [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:26.505383 24854 raft_consensus.cc:2258] T 1a6bd1d77970465487acd1abe91687ad P 5f3e6e0ea51648bb90f831efd3bdd8de [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:26.527846 24854 tablet_server.cc:194] TabletServer@127.24.69.129:0 shutdown complete.
I20240501 20:00:26.540225 24854 tablet_server.cc:178] TabletServer@127.24.69.130:0 shutting down...
I20240501 20:00:26.563266 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:00:26.564010 24854 tablet_replica.cc:330] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df: stopping tablet replica
I20240501 20:00:26.564734 24854 raft_consensus.cc:2229] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:00:26.565989 24854 raft_consensus.cc:2258] T 1a6bd1d77970465487acd1abe91687ad P ab3a5fc4685a45e592318abadee052df [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:26.568802 24854 tablet_replica.cc:330] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df: stopping tablet replica
I20240501 20:00:26.569273 24854 raft_consensus.cc:2229] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:26.569778 24854 raft_consensus.cc:2258] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P ab3a5fc4685a45e592318abadee052df [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:26.609673 24854 tablet_server.cc:194] TabletServer@127.24.69.130:0 shutdown complete.
I20240501 20:00:26.621451 24854 tablet_server.cc:178] TabletServer@127.24.69.131:0 shutting down...
I20240501 20:00:26.645539 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:00:26.646503 24854 tablet_replica.cc:330] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 42f869c3ee6e424093a58fc2ecd6268f: stopping tablet replica
I20240501 20:00:26.647142 24854 raft_consensus.cc:2229] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 42f869c3ee6e424093a58fc2ecd6268f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:26.647959 24854 raft_consensus.cc:2258] T 6bfaaa9fb25a4e18adf32f7339f33fa1 P 42f869c3ee6e424093a58fc2ecd6268f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:26.669265 24854 tablet_server.cc:194] TabletServer@127.24.69.131:0 shutdown complete.
I20240501 20:00:26.680204 24854 tablet_server.cc:178] TabletServer@127.24.69.132:0 shutting down...
I20240501 20:00:26.704751 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:00:26.705345 24854 tablet_replica.cc:330] T 1a6bd1d77970465487acd1abe91687ad P 48fea2b9590b473c87f4db0bf06a88bf: stopping tablet replica
I20240501 20:00:26.705889 24854 raft_consensus.cc:2229] T 1a6bd1d77970465487acd1abe91687ad P 48fea2b9590b473c87f4db0bf06a88bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:26.706594 24854 raft_consensus.cc:2258] T 1a6bd1d77970465487acd1abe91687ad P 48fea2b9590b473c87f4db0bf06a88bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:26.725020 24854 tablet_server.cc:194] TabletServer@127.24.69.132:0 shutdown complete.
I20240501 20:00:26.734582 24854 master.cc:537] Master@127.24.69.190:39931 shutting down...
I20240501 20:00:26.754422 24854 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 69d9e669a0d14218b496566a27f0a70d [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:00:26.755173 24854 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 69d9e669a0d14218b496566a27f0a70d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:26.755652 24854 tablet_replica.cc:330] T 00000000000000000000000000000000 P 69d9e669a0d14218b496566a27f0a70d: stopping tablet replica
I20240501 20:00:26.773429 24854 master.cc:552] Master@127.24.69.190:39931 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (27690 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240501 20:00:26.809300 24854 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.69.190:36229
I20240501 20:00:26.810302 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:26.816542 26962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:26.817469 26965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:26.818243 26963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:26.818958 24854 server_base.cc:1034] running on GCE node
I20240501 20:00:26.819833 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:00:26.820024 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:00:26.820183 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593626820159 us; error 0 us; skew 500 ppm
I20240501 20:00:26.820698 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:00:26.823060 24854 webserver.cc:460] Webserver started at http://127.24.69.190:37689/ using document root <none> and password file <none>
I20240501 20:00:26.823501 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:26.823684 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:26.823951 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:26.825336 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714593576104309-24854-0/minicluster-data/master-0-root/instance:
uuid: "2da9691c0a864ee8a4117b509d2498a3"
format_stamp: "Formatted at 2024-05-01 20:00:26 on dist-test-slave-z8x3"
I20240501 20:00:26.829545 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20240501 20:00:26.832621 26970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:26.833290 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240501 20:00:26.833540 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714593576104309-24854-0/minicluster-data/master-0-root
uuid: "2da9691c0a864ee8a4117b509d2498a3"
format_stamp: "Formatted at 2024-05-01 20:00:26 on dist-test-slave-z8x3"
I20240501 20:00:26.833807 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714593576104309-24854-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714593576104309-24854-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714593576104309-24854-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:26.847396 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:26.848570 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:26.883915 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.190:36229
I20240501 20:00:26.884011 27021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.190:36229 every 8 connection(s)
I20240501 20:00:26.887768 27022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:26.897838 27022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2da9691c0a864ee8a4117b509d2498a3: Bootstrap starting.
I20240501 20:00:26.901795 27022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2da9691c0a864ee8a4117b509d2498a3: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:26.905632 27022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2da9691c0a864ee8a4117b509d2498a3: No bootstrap required, opened a new log
I20240501 20:00:26.907490 27022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2da9691c0a864ee8a4117b509d2498a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da9691c0a864ee8a4117b509d2498a3" member_type: VOTER }
I20240501 20:00:26.907949 27022 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2da9691c0a864ee8a4117b509d2498a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:26.908124 27022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2da9691c0a864ee8a4117b509d2498a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2da9691c0a864ee8a4117b509d2498a3, State: Initialized, Role: FOLLOWER
I20240501 20:00:26.908627 27022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2da9691c0a864ee8a4117b509d2498a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da9691c0a864ee8a4117b509d2498a3" member_type: VOTER }
I20240501 20:00:26.909056 27022 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2da9691c0a864ee8a4117b509d2498a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:00:26.909241 27022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2da9691c0a864ee8a4117b509d2498a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:00:26.909459 27022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2da9691c0a864ee8a4117b509d2498a3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:26.913246 27022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2da9691c0a864ee8a4117b509d2498a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da9691c0a864ee8a4117b509d2498a3" member_type: VOTER }
I20240501 20:00:26.913674 27022 leader_election.cc:302] T 00000000000000000000000000000000 P 2da9691c0a864ee8a4117b509d2498a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2da9691c0a864ee8a4117b509d2498a3; no voters:
I20240501 20:00:26.914641 27022 leader_election.cc:288] T 00000000000000000000000000000000 P 2da9691c0a864ee8a4117b509d2498a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:00:26.915015 27025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2da9691c0a864ee8a4117b509d2498a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:26.916345 27025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2da9691c0a864ee8a4117b509d2498a3 [term 1 LEADER]: Becoming Leader. State: Replica: 2da9691c0a864ee8a4117b509d2498a3, State: Running, Role: LEADER
I20240501 20:00:26.916923 27025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2da9691c0a864ee8a4117b509d2498a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da9691c0a864ee8a4117b509d2498a3" member_type: VOTER }
I20240501 20:00:26.917433 27022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2da9691c0a864ee8a4117b509d2498a3 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:00:26.922303 27027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2da9691c0a864ee8a4117b509d2498a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2da9691c0a864ee8a4117b509d2498a3. Latest consensus state: current_term: 1 leader_uuid: "2da9691c0a864ee8a4117b509d2498a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da9691c0a864ee8a4117b509d2498a3" member_type: VOTER } }
I20240501 20:00:26.923030 27027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2da9691c0a864ee8a4117b509d2498a3 [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:26.922919 27026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2da9691c0a864ee8a4117b509d2498a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2da9691c0a864ee8a4117b509d2498a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da9691c0a864ee8a4117b509d2498a3" member_type: VOTER } }
I20240501 20:00:26.923681 27026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2da9691c0a864ee8a4117b509d2498a3 [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:26.925017 27034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:00:26.929993 27034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:00:26.931836 24854 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 20:00:26.938848 27034 catalog_manager.cc:1347] Generated new cluster ID: 2e0940a7d61b42248736be2d0c361a3a
I20240501 20:00:26.939136 27034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:00:26.960121 27034 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:00:26.961375 27034 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:00:26.977895 27034 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2da9691c0a864ee8a4117b509d2498a3: Generated new TSK 0
I20240501 20:00:26.978436 27034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:00:26.997094 24854 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 0.000179029s
W20240501 20:00:33.123024 27018 debug-util.cc:398] Leaking SignalData structure 0x7b08000db0a0 after lost signal to thread 24857
W20240501 20:00:33.123858 27018 debug-util.cc:398] Leaking SignalData structure 0x7b08000e2280 after lost signal to thread 27021
I20240501 20:00:39.103547 24854 master.cc:537] Master@127.24.69.190:36229 shutting down...
I20240501 20:00:39.131531 24854 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2da9691c0a864ee8a4117b509d2498a3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:00:39.132359 24854 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2da9691c0a864ee8a4117b509d2498a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:39.132714 24854 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2da9691c0a864ee8a4117b509d2498a3: stopping tablet replica
I20240501 20:00:39.167207 24854 master.cc:552] Master@127.24.69.190:36229 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12387 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240501 20:00:39.196635 24854 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.69.190:38157
I20240501 20:00:39.197686 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:39.203799 27045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:39.204416 27046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:39.205250 27048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:39.205708 24854 server_base.cc:1034] running on GCE node
I20240501 20:00:39.207680 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:00:39.207863 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:00:39.208020 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593639207998 us; error 0 us; skew 500 ppm
I20240501 20:00:39.208528 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:00:39.211283 24854 webserver.cc:460] Webserver started at http://127.24.69.190:42093/ using document root <none> and password file <none>
I20240501 20:00:39.211741 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:39.211925 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:39.212181 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:39.213469 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593576104309-24854-0/minicluster-data/master-0-root/instance:
uuid: "eba4b290543644c0a0e7aee75aaff6a3"
format_stamp: "Formatted at 2024-05-01 20:00:39 on dist-test-slave-z8x3"
I20240501 20:00:39.217964 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 20:00:39.221102 27053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:39.221829 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:00:39.222076 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593576104309-24854-0/minicluster-data/master-0-root
uuid: "eba4b290543644c0a0e7aee75aaff6a3"
format_stamp: "Formatted at 2024-05-01 20:00:39 on dist-test-slave-z8x3"
I20240501 20:00:39.222342 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593576104309-24854-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593576104309-24854-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593576104309-24854-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:39.242178 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:39.243341 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:39.279573 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.190:38157
I20240501 20:00:39.279667 27104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.190:38157 every 8 connection(s)
I20240501 20:00:39.283390 27105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:39.294538 27105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba4b290543644c0a0e7aee75aaff6a3: Bootstrap starting.
I20240501 20:00:39.298771 27105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eba4b290543644c0a0e7aee75aaff6a3: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:39.302757 27105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba4b290543644c0a0e7aee75aaff6a3: No bootstrap required, opened a new log
I20240501 20:00:39.304792 27105 raft_consensus.cc:348] T 00000000000000000000000000000000 P eba4b290543644c0a0e7aee75aaff6a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba4b290543644c0a0e7aee75aaff6a3" member_type: VOTER }
I20240501 20:00:39.305256 27105 raft_consensus.cc:374] T 00000000000000000000000000000000 P eba4b290543644c0a0e7aee75aaff6a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:39.305451 27105 raft_consensus.cc:729] T 00000000000000000000000000000000 P eba4b290543644c0a0e7aee75aaff6a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eba4b290543644c0a0e7aee75aaff6a3, State: Initialized, Role: FOLLOWER
I20240501 20:00:39.305899 27105 consensus_queue.cc:260] T 00000000000000000000000000000000 P eba4b290543644c0a0e7aee75aaff6a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba4b290543644c0a0e7aee75aaff6a3" member_type: VOTER }
I20240501 20:00:39.306406 27105 raft_consensus.cc:388] T 00000000000000000000000000000000 P eba4b290543644c0a0e7aee75aaff6a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:00:39.306643 27105 raft_consensus.cc:482] T 00000000000000000000000000000000 P eba4b290543644c0a0e7aee75aaff6a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:00:39.306898 27105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eba4b290543644c0a0e7aee75aaff6a3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:39.311113 27105 raft_consensus.cc:504] T 00000000000000000000000000000000 P eba4b290543644c0a0e7aee75aaff6a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba4b290543644c0a0e7aee75aaff6a3" member_type: VOTER }
I20240501 20:00:39.311564 27105 leader_election.cc:302] T 00000000000000000000000000000000 P eba4b290543644c0a0e7aee75aaff6a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eba4b290543644c0a0e7aee75aaff6a3; no voters:
I20240501 20:00:39.312659 27105 leader_election.cc:288] T 00000000000000000000000000000000 P eba4b290543644c0a0e7aee75aaff6a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:00:39.312968 27108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eba4b290543644c0a0e7aee75aaff6a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:39.314302 27108 raft_consensus.cc:686] T 00000000000000000000000000000000 P eba4b290543644c0a0e7aee75aaff6a3 [term 1 LEADER]: Becoming Leader. State: Replica: eba4b290543644c0a0e7aee75aaff6a3, State: Running, Role: LEADER
I20240501 20:00:39.314990 27108 consensus_queue.cc:237] T 00000000000000000000000000000000 P eba4b290543644c0a0e7aee75aaff6a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba4b290543644c0a0e7aee75aaff6a3" member_type: VOTER }
I20240501 20:00:39.315596 27105 sys_catalog.cc:564] T 00000000000000000000000000000000 P eba4b290543644c0a0e7aee75aaff6a3 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:00:39.318656 27110 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba4b290543644c0a0e7aee75aaff6a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eba4b290543644c0a0e7aee75aaff6a3. Latest consensus state: current_term: 1 leader_uuid: "eba4b290543644c0a0e7aee75aaff6a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba4b290543644c0a0e7aee75aaff6a3" member_type: VOTER } }
I20240501 20:00:39.318575 27109 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba4b290543644c0a0e7aee75aaff6a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eba4b290543644c0a0e7aee75aaff6a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba4b290543644c0a0e7aee75aaff6a3" member_type: VOTER } }
I20240501 20:00:39.319583 27109 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba4b290543644c0a0e7aee75aaff6a3 [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:39.320629 27110 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba4b290543644c0a0e7aee75aaff6a3 [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:39.328150 27115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:00:39.333465 27115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:00:39.334623 24854 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 20:00:39.342028 27115 catalog_manager.cc:1347] Generated new cluster ID: f241f04976b64837a42643ef467f6d13
I20240501 20:00:39.342336 27115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:00:39.359103 27115 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:00:39.360396 27115 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:00:39.371840 27115 catalog_manager.cc:5857] T 00000000000000000000000000000000 P eba4b290543644c0a0e7aee75aaff6a3: Generated new TSK 0
I20240501 20:00:39.372433 27115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:00:39.401002 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:39.407431 27126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:39.408540 27127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:39.409798 27129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:39.410665 24854 server_base.cc:1034] running on GCE node
I20240501 20:00:39.411482 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:00:39.411670 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:00:39.411825 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593639411808 us; error 0 us; skew 500 ppm
I20240501 20:00:39.412379 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:00:39.414604 24854 webserver.cc:460] Webserver started at http://127.24.69.129:34017/ using document root <none> and password file <none>
I20240501 20:00:39.415043 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:39.415222 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:39.415473 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:39.416743 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593576104309-24854-0/minicluster-data/ts-0-root/instance:
uuid: "caba5c656d4e40058f22eba85eaf5bca"
format_stamp: "Formatted at 2024-05-01 20:00:39 on dist-test-slave-z8x3"
I20240501 20:00:39.421025 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20240501 20:00:39.424168 27134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:39.424988 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 20:00:39.425242 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593576104309-24854-0/minicluster-data/ts-0-root
uuid: "caba5c656d4e40058f22eba85eaf5bca"
format_stamp: "Formatted at 2024-05-01 20:00:39 on dist-test-slave-z8x3"
I20240501 20:00:39.425511 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593576104309-24854-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593576104309-24854-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593576104309-24854-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:39.448475 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:39.449589 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:39.451001 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:39.453385 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:39.453593 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:39.453814 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:39.453980 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:39.492487 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.129:41231
I20240501 20:00:39.492587 27196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.129:41231 every 8 connection(s)
I20240501 20:00:39.497104 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:39.505342 27201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:39.506464 27202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:39.508409 27204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:39.509863 24854 server_base.cc:1034] running on GCE node
I20240501 20:00:39.510582 27197 heartbeater.cc:347] Connected to a master server at 127.24.69.190:38157
I20240501 20:00:39.510772 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240501 20:00:39.511009 27197 heartbeater.cc:464] Registering TS with master...
W20240501 20:00:39.511034 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:00:39.511447 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593639511425 us; error 0 us; skew 500 ppm
I20240501 20:00:39.512008 27197 heartbeater.cc:511] Master 127.24.69.190:38157 requested a full tablet report, sending...
I20240501 20:00:39.512217 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:00:39.514298 27070 ts_manager.cc:194] Registered new tserver with Master: caba5c656d4e40058f22eba85eaf5bca (127.24.69.129:41231)
I20240501 20:00:39.514861 24854 webserver.cc:460] Webserver started at http://127.24.69.130:42701/ using document root <none> and password file <none>
I20240501 20:00:39.515327 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:39.515580 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:39.515834 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:39.516311 27070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41312
I20240501 20:00:39.517246 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593576104309-24854-0/minicluster-data/ts-1-root/instance:
uuid: "3079d51ac52d40828bf4358f0da31a7d"
format_stamp: "Formatted at 2024-05-01 20:00:39 on dist-test-slave-z8x3"
I20240501 20:00:39.521816 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 20:00:39.525074 27209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:39.525822 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:00:39.526072 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593576104309-24854-0/minicluster-data/ts-1-root
uuid: "3079d51ac52d40828bf4358f0da31a7d"
format_stamp: "Formatted at 2024-05-01 20:00:39 on dist-test-slave-z8x3"
I20240501 20:00:39.526314 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593576104309-24854-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593576104309-24854-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593576104309-24854-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:39.536909 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:39.538023 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:39.539336 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:39.541513 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:39.541705 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:39.541937 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:39.542094 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:39.580466 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.130:43429
I20240501 20:00:39.580569 27271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.130:43429 every 8 connection(s)
I20240501 20:00:39.585076 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:39.593912 27275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:39.594849 27276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:39.596058 27278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:39.597306 24854 server_base.cc:1034] running on GCE node
I20240501 20:00:39.598114 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:00:39.598316 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:00:39.598335 27272 heartbeater.cc:347] Connected to a master server at 127.24.69.190:38157
I20240501 20:00:39.598558 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593639598542 us; error 0 us; skew 500 ppm
I20240501 20:00:39.598806 27272 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:39.599188 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:00:39.599543 27272 heartbeater.cc:511] Master 127.24.69.190:38157 requested a full tablet report, sending...
I20240501 20:00:39.601503 27070 ts_manager.cc:194] Registered new tserver with Master: 3079d51ac52d40828bf4358f0da31a7d (127.24.69.130:43429)
I20240501 20:00:39.601898 24854 webserver.cc:460] Webserver started at http://127.24.69.131:37595/ using document root <none> and password file <none>
I20240501 20:00:39.602375 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:39.602558 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:39.602811 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:39.602974 27070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41328
I20240501 20:00:39.604169 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593576104309-24854-0/minicluster-data/ts-2-root/instance:
uuid: "76b8a109241346bd98a9bf9dff3565d4"
format_stamp: "Formatted at 2024-05-01 20:00:39 on dist-test-slave-z8x3"
I20240501 20:00:39.608529 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20240501 20:00:39.611663 27283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:39.612425 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 20:00:39.612687 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593576104309-24854-0/minicluster-data/ts-2-root
uuid: "76b8a109241346bd98a9bf9dff3565d4"
format_stamp: "Formatted at 2024-05-01 20:00:39 on dist-test-slave-z8x3"
I20240501 20:00:39.612967 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593576104309-24854-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593576104309-24854-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593576104309-24854-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:39.641131 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:39.642270 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:39.643638 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:39.645843 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:39.646035 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:39.646262 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:39.646414 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:39.684098 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.131:36193
I20240501 20:00:39.684190 27345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.131:36193 every 8 connection(s)
I20240501 20:00:39.697095 27346 heartbeater.cc:347] Connected to a master server at 127.24.69.190:38157
I20240501 20:00:39.697450 27346 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:39.698130 27346 heartbeater.cc:511] Master 127.24.69.190:38157 requested a full tablet report, sending...
I20240501 20:00:39.699972 27070 ts_manager.cc:194] Registered new tserver with Master: 76b8a109241346bd98a9bf9dff3565d4 (127.24.69.131:36193)
I20240501 20:00:39.700896 24854 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013721646s
I20240501 20:00:39.701567 27070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41342
I20240501 20:00:40.519042 27197 heartbeater.cc:503] Master 127.24.69.190:38157 was elected leader, sending a full tablet report...
I20240501 20:00:40.605315 27272 heartbeater.cc:503] Master 127.24.69.190:38157 was elected leader, sending a full tablet report...
I20240501 20:00:40.704095 27346 heartbeater.cc:503] Master 127.24.69.190:38157 was elected leader, sending a full tablet report...
I20240501 20:00:41.331153 27123 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:43.331977 27123 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:45.382081 27123 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240501 20:00:46.345338 27101 debug-util.cc:398] Leaking SignalData structure 0x7b08000c1240 after lost signal to thread 24857
W20240501 20:00:46.346318 27101 debug-util.cc:398] Leaking SignalData structure 0x7b080023ab80 after lost signal to thread 27104
W20240501 20:00:46.347254 27101 debug-util.cc:398] Leaking SignalData structure 0x7b080023f000 after lost signal to thread 27196
W20240501 20:00:46.348227 27101 debug-util.cc:398] Leaking SignalData structure 0x7b08000c2d00 after lost signal to thread 27271
W20240501 20:00:46.349086 27101 debug-util.cc:398] Leaking SignalData structure 0x7b0800080920 after lost signal to thread 27345
W20240501 20:00:47.008545 27193 debug-util.cc:398] Leaking SignalData structure 0x7b08000bcb80 after lost signal to thread 24857
W20240501 20:00:47.010221 27193 debug-util.cc:398] Leaking SignalData structure 0x7b0800098e60 after lost signal to thread 27104
W20240501 20:00:47.011868 27193 debug-util.cc:398] Leaking SignalData structure 0x7b08000a17a0 after lost signal to thread 27196
W20240501 20:00:47.013355 27193 debug-util.cc:398] Leaking SignalData structure 0x7b08001a26c0 after lost signal to thread 27271
W20240501 20:00:47.014717 27193 debug-util.cc:398] Leaking SignalData structure 0x7b080004d1e0 after lost signal to thread 27345
I20240501 20:00:47.383612 27123 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:49.384773 27123 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:51.385654 27123 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:52.151042 24854 tablet_server.cc:178] TabletServer@127.24.69.129:0 shutting down...
I20240501 20:00:52.177639 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:00:52.210759 24854 tablet_server.cc:194] TabletServer@127.24.69.129:0 shutdown complete.
I20240501 20:00:52.220453 24854 tablet_server.cc:178] TabletServer@127.24.69.130:0 shutting down...
I20240501 20:00:52.240650 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:00:52.256340 24854 tablet_server.cc:194] TabletServer@127.24.69.130:0 shutdown complete.
I20240501 20:00:52.264091 24854 tablet_server.cc:178] TabletServer@127.24.69.131:0 shutting down...
I20240501 20:00:52.282706 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:00:52.298629 24854 tablet_server.cc:194] TabletServer@127.24.69.131:0 shutdown complete.
I20240501 20:00:52.306963 24854 master.cc:537] Master@127.24.69.190:38157 shutting down...
I20240501 20:00:52.323570 24854 raft_consensus.cc:2229] T 00000000000000000000000000000000 P eba4b290543644c0a0e7aee75aaff6a3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:00:52.324334 24854 raft_consensus.cc:2258] T 00000000000000000000000000000000 P eba4b290543644c0a0e7aee75aaff6a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:52.324654 24854 tablet_replica.cc:330] T 00000000000000000000000000000000 P eba4b290543644c0a0e7aee75aaff6a3: stopping tablet replica
I20240501 20:00:52.357028 24854 master.cc:552] Master@127.24.69.190:38157 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (13189 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240501 20:00:52.386040 24854 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.69.190:35175
I20240501 20:00:52.387115 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:52.393198 27351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:52.393776 27352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:52.394608 27354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:52.395296 24854 server_base.cc:1034] running on GCE node
I20240501 20:00:52.397159 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:00:52.397346 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:00:52.397485 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593652397468 us; error 0 us; skew 500 ppm
I20240501 20:00:52.397966 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:00:52.404536 24854 webserver.cc:460] Webserver started at http://127.24.69.190:37967/ using document root <none> and password file <none>
I20240501 20:00:52.404981 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:52.405148 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:52.405414 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:52.406607 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593576104309-24854-0/minicluster-data/master-0-root/instance:
uuid: "92b8533d4bda4aaba471bc213af6ef1b"
format_stamp: "Formatted at 2024-05-01 20:00:52 on dist-test-slave-z8x3"
I20240501 20:00:52.411072 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20240501 20:00:52.414240 27359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:52.414937 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:00:52.415190 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593576104309-24854-0/minicluster-data/master-0-root
uuid: "92b8533d4bda4aaba471bc213af6ef1b"
format_stamp: "Formatted at 2024-05-01 20:00:52 on dist-test-slave-z8x3"
I20240501 20:00:52.415449 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593576104309-24854-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593576104309-24854-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593576104309-24854-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:52.435995 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:52.437103 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:52.473030 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.190:35175
I20240501 20:00:52.473134 27410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.190:35175 every 8 connection(s)
I20240501 20:00:52.476701 27411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:52.487454 27411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92b8533d4bda4aaba471bc213af6ef1b: Bootstrap starting.
I20240501 20:00:52.491868 27411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92b8533d4bda4aaba471bc213af6ef1b: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:52.495736 27411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92b8533d4bda4aaba471bc213af6ef1b: No bootstrap required, opened a new log
I20240501 20:00:52.497679 27411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92b8533d4bda4aaba471bc213af6ef1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b8533d4bda4aaba471bc213af6ef1b" member_type: VOTER }
I20240501 20:00:52.498118 27411 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92b8533d4bda4aaba471bc213af6ef1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:52.498303 27411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92b8533d4bda4aaba471bc213af6ef1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b8533d4bda4aaba471bc213af6ef1b, State: Initialized, Role: FOLLOWER
I20240501 20:00:52.498775 27411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92b8533d4bda4aaba471bc213af6ef1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b8533d4bda4aaba471bc213af6ef1b" member_type: VOTER }
I20240501 20:00:52.499194 27411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92b8533d4bda4aaba471bc213af6ef1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:00:52.499388 27411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92b8533d4bda4aaba471bc213af6ef1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:00:52.499599 27411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92b8533d4bda4aaba471bc213af6ef1b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:52.503793 27411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92b8533d4bda4aaba471bc213af6ef1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b8533d4bda4aaba471bc213af6ef1b" member_type: VOTER }
I20240501 20:00:52.504231 27411 leader_election.cc:302] T 00000000000000000000000000000000 P 92b8533d4bda4aaba471bc213af6ef1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92b8533d4bda4aaba471bc213af6ef1b; no voters:
I20240501 20:00:52.505261 27411 leader_election.cc:288] T 00000000000000000000000000000000 P 92b8533d4bda4aaba471bc213af6ef1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:00:52.505647 27414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92b8533d4bda4aaba471bc213af6ef1b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:52.506961 27414 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92b8533d4bda4aaba471bc213af6ef1b [term 1 LEADER]: Becoming Leader. State: Replica: 92b8533d4bda4aaba471bc213af6ef1b, State: Running, Role: LEADER
I20240501 20:00:52.507625 27414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92b8533d4bda4aaba471bc213af6ef1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b8533d4bda4aaba471bc213af6ef1b" member_type: VOTER }
I20240501 20:00:52.507997 27411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92b8533d4bda4aaba471bc213af6ef1b [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:00:52.512789 27415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92b8533d4bda4aaba471bc213af6ef1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92b8533d4bda4aaba471bc213af6ef1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b8533d4bda4aaba471bc213af6ef1b" member_type: VOTER } }
I20240501 20:00:52.513118 27416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92b8533d4bda4aaba471bc213af6ef1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92b8533d4bda4aaba471bc213af6ef1b. Latest consensus state: current_term: 1 leader_uuid: "92b8533d4bda4aaba471bc213af6ef1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b8533d4bda4aaba471bc213af6ef1b" member_type: VOTER } }
I20240501 20:00:52.513608 27415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92b8533d4bda4aaba471bc213af6ef1b [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:52.513942 27416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92b8533d4bda4aaba471bc213af6ef1b [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:52.522699 27422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:00:52.527392 27422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:00:52.528463 24854 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 20:00:52.535411 27422 catalog_manager.cc:1347] Generated new cluster ID: b88b9c6dab794079b10e6433d1cbdddc
I20240501 20:00:52.535681 27422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:00:52.550845 27422 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:00:52.552059 27422 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:00:52.563400 27422 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 92b8533d4bda4aaba471bc213af6ef1b: Generated new TSK 0
I20240501 20:00:52.563941 27422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:00:52.594826 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:52.601315 27432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:52.602160 27433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:52.603245 27435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:52.604283 24854 server_base.cc:1034] running on GCE node
I20240501 20:00:52.605116 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:00:52.605309 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:00:52.605477 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593652605463 us; error 0 us; skew 500 ppm
I20240501 20:00:52.606025 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:00:52.608399 24854 webserver.cc:460] Webserver started at http://127.24.69.129:42033/ using document root <none> and password file <none>
I20240501 20:00:52.608846 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:52.609038 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:52.609297 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:52.610401 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593576104309-24854-0/minicluster-data/ts-0-root/instance:
uuid: "67e85b2646ea416ebe5acd33c3007033"
format_stamp: "Formatted at 2024-05-01 20:00:52 on dist-test-slave-z8x3"
I20240501 20:00:52.614836 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 20:00:52.618078 27440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:52.618861 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:00:52.619113 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593576104309-24854-0/minicluster-data/ts-0-root
uuid: "67e85b2646ea416ebe5acd33c3007033"
format_stamp: "Formatted at 2024-05-01 20:00:52 on dist-test-slave-z8x3"
I20240501 20:00:52.619395 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593576104309-24854-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593576104309-24854-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593576104309-24854-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:52.649158 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:52.650234 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:52.651609 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:52.653790 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:52.654004 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:52.654214 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:52.654378 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:52.692867 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.129:42341
I20240501 20:00:52.692957 27502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.129:42341 every 8 connection(s)
I20240501 20:00:52.698621 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:52.705098 27507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:52.706203 27508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:52.708681 27510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:52.709672 24854 server_base.cc:1034] running on GCE node
I20240501 20:00:52.710065 27503 heartbeater.cc:347] Connected to a master server at 127.24.69.190:35175
I20240501 20:00:52.710397 27503 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:52.710505 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:00:52.710732 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:00:52.710888 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593652710866 us; error 0 us; skew 500 ppm
I20240501 20:00:52.711124 27503 heartbeater.cc:511] Master 127.24.69.190:35175 requested a full tablet report, sending...
I20240501 20:00:52.711460 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:00:52.713232 27376 ts_manager.cc:194] Registered new tserver with Master: 67e85b2646ea416ebe5acd33c3007033 (127.24.69.129:42341)
I20240501 20:00:52.714130 24854 webserver.cc:460] Webserver started at http://127.24.69.130:46717/ using document root <none> and password file <none>
I20240501 20:00:52.714744 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:52.714985 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:52.715291 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:52.715795 27376 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47226
I20240501 20:00:52.716861 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593576104309-24854-0/minicluster-data/ts-1-root/instance:
uuid: "63da28a4423e4abe844d40889b1b2652"
format_stamp: "Formatted at 2024-05-01 20:00:52 on dist-test-slave-z8x3"
I20240501 20:00:52.721310 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240501 20:00:52.724164 27515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:52.724885 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:00:52.725137 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593576104309-24854-0/minicluster-data/ts-1-root
uuid: "63da28a4423e4abe844d40889b1b2652"
format_stamp: "Formatted at 2024-05-01 20:00:52 on dist-test-slave-z8x3"
I20240501 20:00:52.725397 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593576104309-24854-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593576104309-24854-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593576104309-24854-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:52.761257 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:52.762332 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:52.763711 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:52.765870 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:52.766068 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:52.766278 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:52.766427 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:52.804522 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.130:34239
I20240501 20:00:52.804617 27577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.130:34239 every 8 connection(s)
I20240501 20:00:52.809090 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:52.816017 27581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:52.817009 27582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:52.819746 27584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:52.820827 24854 server_base.cc:1034] running on GCE node
I20240501 20:00:52.821295 27578 heartbeater.cc:347] Connected to a master server at 127.24.69.190:35175
I20240501 20:00:52.821720 27578 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:52.821810 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:00:52.822095 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:00:52.822299 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593652822279 us; error 0 us; skew 500 ppm
I20240501 20:00:52.822631 27578 heartbeater.cc:511] Master 127.24.69.190:35175 requested a full tablet report, sending...
I20240501 20:00:52.823060 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:00:52.824923 27376 ts_manager.cc:194] Registered new tserver with Master: 63da28a4423e4abe844d40889b1b2652 (127.24.69.130:34239)
I20240501 20:00:52.825650 24854 webserver.cc:460] Webserver started at http://127.24.69.131:40211/ using document root <none> and password file <none>
I20240501 20:00:52.826133 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:52.826344 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:52.826462 27376 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47234
I20240501 20:00:52.826618 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:52.827982 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593576104309-24854-0/minicluster-data/ts-2-root/instance:
uuid: "7ffcddf6f02748018bea4659244b21ef"
format_stamp: "Formatted at 2024-05-01 20:00:52 on dist-test-slave-z8x3"
I20240501 20:00:52.832123 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240501 20:00:52.835202 27589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:52.835880 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240501 20:00:52.836138 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593576104309-24854-0/minicluster-data/ts-2-root
uuid: "7ffcddf6f02748018bea4659244b21ef"
format_stamp: "Formatted at 2024-05-01 20:00:52 on dist-test-slave-z8x3"
I20240501 20:00:52.836459 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593576104309-24854-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593576104309-24854-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593576104309-24854-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:52.852783 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:52.853863 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:52.855245 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:52.857430 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:52.857626 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:52.857846 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:52.857996 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:52.895311 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.131:34609
I20240501 20:00:52.895406 27651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.131:34609 every 8 connection(s)
I20240501 20:00:52.899859 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:52.906761 27655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:52.909543 27656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:52.910612 24854 server_base.cc:1034] running on GCE node
W20240501 20:00:52.911226 27658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:52.911612 27652 heartbeater.cc:347] Connected to a master server at 127.24.69.190:35175
I20240501 20:00:52.912072 27652 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:52.912240 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:00:52.912545 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:00:52.912755 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593652912738 us; error 0 us; skew 500 ppm
I20240501 20:00:52.912967 27652 heartbeater.cc:511] Master 127.24.69.190:35175 requested a full tablet report, sending...
I20240501 20:00:52.913491 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:00:52.915331 27376 ts_manager.cc:194] Registered new tserver with Master: 7ffcddf6f02748018bea4659244b21ef (127.24.69.131:34609)
I20240501 20:00:52.916514 24854 webserver.cc:460] Webserver started at http://127.24.69.132:41147/ using document root <none> and password file <none>
I20240501 20:00:52.917127 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:52.917217 27376 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47246
I20240501 20:00:52.917414 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:52.917769 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:52.919205 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593576104309-24854-0/minicluster-data/ts-3-root/instance:
uuid: "a7d2ee387641454987769bae8dcd4727"
format_stamp: "Formatted at 2024-05-01 20:00:52 on dist-test-slave-z8x3"
I20240501 20:00:52.923516 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 20:00:52.926515 27663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:52.927249 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:00:52.927510 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593576104309-24854-0/minicluster-data/ts-3-root
uuid: "a7d2ee387641454987769bae8dcd4727"
format_stamp: "Formatted at 2024-05-01 20:00:52 on dist-test-slave-z8x3"
I20240501 20:00:52.927727 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593576104309-24854-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593576104309-24854-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593576104309-24854-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:52.942831 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:52.943900 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:52.945314 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:52.947556 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:52.947746 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:52.948022 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:52.948225 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:52.987320 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.132:44121
I20240501 20:00:52.987426 27725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.132:44121 every 8 connection(s)
I20240501 20:00:53.000986 27726 heartbeater.cc:347] Connected to a master server at 127.24.69.190:35175
I20240501 20:00:53.001310 27726 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:53.002038 27726 heartbeater.cc:511] Master 127.24.69.190:35175 requested a full tablet report, sending...
I20240501 20:00:53.003747 27376 ts_manager.cc:194] Registered new tserver with Master: a7d2ee387641454987769bae8dcd4727 (127.24.69.132:44121)
I20240501 20:00:53.004084 24854 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.013674994s
I20240501 20:00:53.005609 27376 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47256
I20240501 20:00:53.535439 24854 test_util.cc:268] Using random seed: -240700744
I20240501 20:00:53.557011 27376 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47270:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 20:00:53.601717 27543 tablet_service.cc:1467] Processing CreateTablet for tablet 4f47a1e240a24140a41e107d60ebbed5 (DEFAULT_TABLE table=test-workload [id=8d291f1ae2574ac0a46a7a927ec55eaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:00:53.603026 27543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f47a1e240a24140a41e107d60ebbed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:53.605190 27691 tablet_service.cc:1467] Processing CreateTablet for tablet 4f47a1e240a24140a41e107d60ebbed5 (DEFAULT_TABLE table=test-workload [id=8d291f1ae2574ac0a46a7a927ec55eaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:00:53.606413 27691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f47a1e240a24140a41e107d60ebbed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:53.607782 27617 tablet_service.cc:1467] Processing CreateTablet for tablet 4f47a1e240a24140a41e107d60ebbed5 (DEFAULT_TABLE table=test-workload [id=8d291f1ae2574ac0a46a7a927ec55eaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:00:53.609004 27617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f47a1e240a24140a41e107d60ebbed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:53.627893 27747 tablet_bootstrap.cc:492] T 4f47a1e240a24140a41e107d60ebbed5 P a7d2ee387641454987769bae8dcd4727: Bootstrap starting.
I20240501 20:00:53.628211 27748 tablet_bootstrap.cc:492] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652: Bootstrap starting.
I20240501 20:00:53.629366 27749 tablet_bootstrap.cc:492] T 4f47a1e240a24140a41e107d60ebbed5 P 7ffcddf6f02748018bea4659244b21ef: Bootstrap starting.
I20240501 20:00:53.633663 27748 tablet_bootstrap.cc:654] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:53.635481 27749 tablet_bootstrap.cc:654] T 4f47a1e240a24140a41e107d60ebbed5 P 7ffcddf6f02748018bea4659244b21ef: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:53.635481 27747 tablet_bootstrap.cc:654] T 4f47a1e240a24140a41e107d60ebbed5 P a7d2ee387641454987769bae8dcd4727: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:53.639073 27748 tablet_bootstrap.cc:492] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652: No bootstrap required, opened a new log
I20240501 20:00:53.639593 27748 ts_tablet_manager.cc:1367] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20240501 20:00:53.641299 27749 tablet_bootstrap.cc:492] T 4f47a1e240a24140a41e107d60ebbed5 P 7ffcddf6f02748018bea4659244b21ef: No bootstrap required, opened a new log
I20240501 20:00:53.641769 27749 ts_tablet_manager.cc:1367] T 4f47a1e240a24140a41e107d60ebbed5 P 7ffcddf6f02748018bea4659244b21ef: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20240501 20:00:53.642401 27747 tablet_bootstrap.cc:492] T 4f47a1e240a24140a41e107d60ebbed5 P a7d2ee387641454987769bae8dcd4727: No bootstrap required, opened a new log
I20240501 20:00:53.642134 27748 raft_consensus.cc:348] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63da28a4423e4abe844d40889b1b2652" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 34239 } } peers { permanent_uuid: "a7d2ee387641454987769bae8dcd4727" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 44121 } } peers { permanent_uuid: "7ffcddf6f02748018bea4659244b21ef" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 34609 } }
I20240501 20:00:53.642838 27747 ts_tablet_manager.cc:1367] T 4f47a1e240a24140a41e107d60ebbed5 P a7d2ee387641454987769bae8dcd4727: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20240501 20:00:53.642913 27748 raft_consensus.cc:374] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:53.643335 27748 raft_consensus.cc:729] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63da28a4423e4abe844d40889b1b2652, State: Initialized, Role: FOLLOWER
I20240501 20:00:53.644119 27748 consensus_queue.cc:260] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63da28a4423e4abe844d40889b1b2652" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 34239 } } peers { permanent_uuid: "a7d2ee387641454987769bae8dcd4727" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 44121 } } peers { permanent_uuid: "7ffcddf6f02748018bea4659244b21ef" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 34609 } }
I20240501 20:00:53.644402 27749 raft_consensus.cc:348] T 4f47a1e240a24140a41e107d60ebbed5 P 7ffcddf6f02748018bea4659244b21ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63da28a4423e4abe844d40889b1b2652" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 34239 } } peers { permanent_uuid: "a7d2ee387641454987769bae8dcd4727" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 44121 } } peers { permanent_uuid: "7ffcddf6f02748018bea4659244b21ef" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 34609 } }
I20240501 20:00:53.645160 27749 raft_consensus.cc:374] T 4f47a1e240a24140a41e107d60ebbed5 P 7ffcddf6f02748018bea4659244b21ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:53.645467 27749 raft_consensus.cc:729] T 4f47a1e240a24140a41e107d60ebbed5 P 7ffcddf6f02748018bea4659244b21ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ffcddf6f02748018bea4659244b21ef, State: Initialized, Role: FOLLOWER
I20240501 20:00:53.645543 27747 raft_consensus.cc:348] T 4f47a1e240a24140a41e107d60ebbed5 P a7d2ee387641454987769bae8dcd4727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63da28a4423e4abe844d40889b1b2652" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 34239 } } peers { permanent_uuid: "a7d2ee387641454987769bae8dcd4727" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 44121 } } peers { permanent_uuid: "7ffcddf6f02748018bea4659244b21ef" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 34609 } }
I20240501 20:00:53.646340 27747 raft_consensus.cc:374] T 4f47a1e240a24140a41e107d60ebbed5 P a7d2ee387641454987769bae8dcd4727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:53.646735 27747 raft_consensus.cc:729] T 4f47a1e240a24140a41e107d60ebbed5 P a7d2ee387641454987769bae8dcd4727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7d2ee387641454987769bae8dcd4727, State: Initialized, Role: FOLLOWER
I20240501 20:00:53.646276 27749 consensus_queue.cc:260] T 4f47a1e240a24140a41e107d60ebbed5 P 7ffcddf6f02748018bea4659244b21ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63da28a4423e4abe844d40889b1b2652" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 34239 } } peers { permanent_uuid: "a7d2ee387641454987769bae8dcd4727" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 44121 } } peers { permanent_uuid: "7ffcddf6f02748018bea4659244b21ef" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 34609 } }
I20240501 20:00:53.647527 27747 consensus_queue.cc:260] T 4f47a1e240a24140a41e107d60ebbed5 P a7d2ee387641454987769bae8dcd4727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63da28a4423e4abe844d40889b1b2652" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 34239 } } peers { permanent_uuid: "a7d2ee387641454987769bae8dcd4727" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 44121 } } peers { permanent_uuid: "7ffcddf6f02748018bea4659244b21ef" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 34609 } }
I20240501 20:00:53.649215 27652 heartbeater.cc:503] Master 127.24.69.190:35175 was elected leader, sending a full tablet report...
W20240501 20:00:53.649585 27653 tablet.cc:2369] T 4f47a1e240a24140a41e107d60ebbed5 P 7ffcddf6f02748018bea4659244b21ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:00:53.650972 27749 ts_tablet_manager.cc:1398] T 4f47a1e240a24140a41e107d60ebbed5 P 7ffcddf6f02748018bea4659244b21ef: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20240501 20:00:53.652946 27578 heartbeater.cc:503] Master 127.24.69.190:35175 was elected leader, sending a full tablet report...
I20240501 20:00:53.653384 27748 ts_tablet_manager.cc:1398] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652: Time spent starting tablet: real 0.013s user 0.006s sys 0.008s
I20240501 20:00:53.653999 27726 heartbeater.cc:503] Master 127.24.69.190:35175 was elected leader, sending a full tablet report...
I20240501 20:00:53.656456 27747 ts_tablet_manager.cc:1398] T 4f47a1e240a24140a41e107d60ebbed5 P a7d2ee387641454987769bae8dcd4727: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20240501 20:00:53.718222 27503 heartbeater.cc:503] Master 127.24.69.190:35175 was elected leader, sending a full tablet report...
I20240501 20:00:53.771812 27753 raft_consensus.cc:482] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:53.772315 27753 raft_consensus.cc:504] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63da28a4423e4abe844d40889b1b2652" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 34239 } } peers { permanent_uuid: "a7d2ee387641454987769bae8dcd4727" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 44121 } } peers { permanent_uuid: "7ffcddf6f02748018bea4659244b21ef" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 34609 } }
I20240501 20:00:53.774410 27753 leader_election.cc:288] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7d2ee387641454987769bae8dcd4727 (127.24.69.132:44121), 7ffcddf6f02748018bea4659244b21ef (127.24.69.131:34609)
I20240501 20:00:53.784041 27701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f47a1e240a24140a41e107d60ebbed5" candidate_uuid: "63da28a4423e4abe844d40889b1b2652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7d2ee387641454987769bae8dcd4727" is_pre_election: true
I20240501 20:00:53.784672 27627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f47a1e240a24140a41e107d60ebbed5" candidate_uuid: "63da28a4423e4abe844d40889b1b2652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ffcddf6f02748018bea4659244b21ef" is_pre_election: true
I20240501 20:00:53.784984 27701 raft_consensus.cc:2454] T 4f47a1e240a24140a41e107d60ebbed5 P a7d2ee387641454987769bae8dcd4727 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63da28a4423e4abe844d40889b1b2652 in term 0.
I20240501 20:00:53.785385 27627 raft_consensus.cc:2454] T 4f47a1e240a24140a41e107d60ebbed5 P 7ffcddf6f02748018bea4659244b21ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63da28a4423e4abe844d40889b1b2652 in term 0.
I20240501 20:00:53.786221 27516 leader_election.cc:302] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63da28a4423e4abe844d40889b1b2652, a7d2ee387641454987769bae8dcd4727; no voters:
I20240501 20:00:53.786900 27753 raft_consensus.cc:2781] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:53.787235 27753 raft_consensus.cc:482] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:53.787492 27753 raft_consensus.cc:3037] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:53.792424 27753 raft_consensus.cc:504] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63da28a4423e4abe844d40889b1b2652" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 34239 } } peers { permanent_uuid: "a7d2ee387641454987769bae8dcd4727" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 44121 } } peers { permanent_uuid: "7ffcddf6f02748018bea4659244b21ef" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 34609 } }
I20240501 20:00:53.793884 27753 leader_election.cc:288] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652 [CANDIDATE]: Term 1 election: Requested vote from peers a7d2ee387641454987769bae8dcd4727 (127.24.69.132:44121), 7ffcddf6f02748018bea4659244b21ef (127.24.69.131:34609)
I20240501 20:00:53.794534 27701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f47a1e240a24140a41e107d60ebbed5" candidate_uuid: "63da28a4423e4abe844d40889b1b2652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7d2ee387641454987769bae8dcd4727"
I20240501 20:00:53.794775 27627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f47a1e240a24140a41e107d60ebbed5" candidate_uuid: "63da28a4423e4abe844d40889b1b2652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ffcddf6f02748018bea4659244b21ef"
I20240501 20:00:53.795130 27701 raft_consensus.cc:3037] T 4f47a1e240a24140a41e107d60ebbed5 P a7d2ee387641454987769bae8dcd4727 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:53.795290 27627 raft_consensus.cc:3037] T 4f47a1e240a24140a41e107d60ebbed5 P 7ffcddf6f02748018bea4659244b21ef [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:53.801976 27627 raft_consensus.cc:2454] T 4f47a1e240a24140a41e107d60ebbed5 P 7ffcddf6f02748018bea4659244b21ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63da28a4423e4abe844d40889b1b2652 in term 1.
I20240501 20:00:53.801976 27701 raft_consensus.cc:2454] T 4f47a1e240a24140a41e107d60ebbed5 P a7d2ee387641454987769bae8dcd4727 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63da28a4423e4abe844d40889b1b2652 in term 1.
I20240501 20:00:53.803289 27518 leader_election.cc:302] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63da28a4423e4abe844d40889b1b2652, 7ffcddf6f02748018bea4659244b21ef; no voters:
I20240501 20:00:53.803943 27753 raft_consensus.cc:2781] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:53.804970 27753 raft_consensus.cc:686] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652 [term 1 LEADER]: Becoming Leader. State: Replica: 63da28a4423e4abe844d40889b1b2652, State: Running, Role: LEADER
I20240501 20:00:53.805562 27753 consensus_queue.cc:237] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63da28a4423e4abe844d40889b1b2652" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 34239 } } peers { permanent_uuid: "a7d2ee387641454987769bae8dcd4727" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 44121 } } peers { permanent_uuid: "7ffcddf6f02748018bea4659244b21ef" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 34609 } }
I20240501 20:00:53.812687 27374 catalog_manager.cc:5484] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63da28a4423e4abe844d40889b1b2652 (127.24.69.130). New cstate: current_term: 1 leader_uuid: "63da28a4423e4abe844d40889b1b2652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63da28a4423e4abe844d40889b1b2652" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 34239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7d2ee387641454987769bae8dcd4727" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 44121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ffcddf6f02748018bea4659244b21ef" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 34609 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:54.369062 27760 consensus_queue.cc:1035] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ffcddf6f02748018bea4659244b21ef" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 34609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:54.384003 27760 consensus_queue.cc:1035] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7d2ee387641454987769bae8dcd4727" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 44121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:54.524688 27429 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:54.526459 27429 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:54.526873 27429 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:56.527843 27429 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:56.529255 27429 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:56.529582 27429 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:58.530644 27429 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:58.532032 27429 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:58.532335 27429 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:01:00.533198 27429 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:00.534402 27429 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:01:00.534655 27429 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:01:02.535491 27429 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:02.536912 27429 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:01:02.537173 27429 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:01:04.538048 27429 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:04.539268 27429 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:01:04.539537 27429 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:01:04.898584 24854 tablet_server.cc:178] TabletServer@127.24.69.129:0 shutting down...
I20240501 20:01:04.922356 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:01:04.939332 24854 tablet_server.cc:194] TabletServer@127.24.69.129:0 shutdown complete.
I20240501 20:01:04.948949 24854 tablet_server.cc:178] TabletServer@127.24.69.130:0 shutting down...
I20240501 20:01:04.970225 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:01:04.970844 24854 tablet_replica.cc:330] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652: stopping tablet replica
I20240501 20:01:04.971386 24854 raft_consensus.cc:2229] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:01:04.972280 24854 raft_consensus.cc:2258] T 4f47a1e240a24140a41e107d60ebbed5 P 63da28a4423e4abe844d40889b1b2652 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:04.991523 24854 tablet_server.cc:194] TabletServer@127.24.69.130:0 shutdown complete.
I20240501 20:01:05.001736 24854 tablet_server.cc:178] TabletServer@127.24.69.131:0 shutting down...
I20240501 20:01:05.022393 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:01:05.023095 24854 tablet_replica.cc:330] T 4f47a1e240a24140a41e107d60ebbed5 P 7ffcddf6f02748018bea4659244b21ef: stopping tablet replica
I20240501 20:01:05.023721 24854 raft_consensus.cc:2229] T 4f47a1e240a24140a41e107d60ebbed5 P 7ffcddf6f02748018bea4659244b21ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:05.024199 24854 raft_consensus.cc:2258] T 4f47a1e240a24140a41e107d60ebbed5 P 7ffcddf6f02748018bea4659244b21ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:05.042644 24854 tablet_server.cc:194] TabletServer@127.24.69.131:0 shutdown complete.
I20240501 20:01:05.052179 24854 tablet_server.cc:178] TabletServer@127.24.69.132:0 shutting down...
I20240501 20:01:05.070770 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:01:05.071419 24854 tablet_replica.cc:330] T 4f47a1e240a24140a41e107d60ebbed5 P a7d2ee387641454987769bae8dcd4727: stopping tablet replica
I20240501 20:01:05.072016 24854 raft_consensus.cc:2229] T 4f47a1e240a24140a41e107d60ebbed5 P a7d2ee387641454987769bae8dcd4727 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:05.072528 24854 raft_consensus.cc:2258] T 4f47a1e240a24140a41e107d60ebbed5 P a7d2ee387641454987769bae8dcd4727 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:05.090925 24854 tablet_server.cc:194] TabletServer@127.24.69.132:0 shutdown complete.
I20240501 20:01:05.100374 24854 master.cc:537] Master@127.24.69.190:35175 shutting down...
I20240501 20:01:05.119330 24854 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 92b8533d4bda4aaba471bc213af6ef1b [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:01:05.119977 24854 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 92b8533d4bda4aaba471bc213af6ef1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:05.120342 24854 tablet_replica.cc:330] T 00000000000000000000000000000000 P 92b8533d4bda4aaba471bc213af6ef1b: stopping tablet replica
I20240501 20:01:05.138168 24854 master.cc:552] Master@127.24.69.190:35175 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12787 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240501 20:01:05.173310 24854 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.69.190:36331
I20240501 20:01:05.174345 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:05.180363 27782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:05.180905 27783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:05.181455 27785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:05.182158 24854 server_base.cc:1034] running on GCE node
I20240501 20:01:05.182859 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:01:05.183041 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:05.183192 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593665183174 us; error 0 us; skew 500 ppm
I20240501 20:01:05.183676 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:05.190125 24854 webserver.cc:460] Webserver started at http://127.24.69.190:46349/ using document root <none> and password file <none>
I20240501 20:01:05.190604 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:05.190762 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:05.191044 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:05.192240 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593576104309-24854-0/minicluster-data/master-0-root/instance:
uuid: "47df0b1c907e43b8a88e258dd33b74b3"
format_stamp: "Formatted at 2024-05-01 20:01:05 on dist-test-slave-z8x3"
I20240501 20:01:05.196557 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20240501 20:01:05.199594 27790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:05.200284 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 20:01:05.200533 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593576104309-24854-0/minicluster-data/master-0-root
uuid: "47df0b1c907e43b8a88e258dd33b74b3"
format_stamp: "Formatted at 2024-05-01 20:01:05 on dist-test-slave-z8x3"
I20240501 20:01:05.200809 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593576104309-24854-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593576104309-24854-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593576104309-24854-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:05.216521 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:05.217634 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:05.252871 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.190:36331
I20240501 20:01:05.252962 27841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.190:36331 every 8 connection(s)
I20240501 20:01:05.256632 27842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:05.266918 27842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47df0b1c907e43b8a88e258dd33b74b3: Bootstrap starting.
I20240501 20:01:05.270965 27842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47df0b1c907e43b8a88e258dd33b74b3: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:05.274833 27842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47df0b1c907e43b8a88e258dd33b74b3: No bootstrap required, opened a new log
I20240501 20:01:05.276907 27842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47df0b1c907e43b8a88e258dd33b74b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47df0b1c907e43b8a88e258dd33b74b3" member_type: VOTER }
I20240501 20:01:05.277354 27842 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47df0b1c907e43b8a88e258dd33b74b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:05.277529 27842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47df0b1c907e43b8a88e258dd33b74b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47df0b1c907e43b8a88e258dd33b74b3, State: Initialized, Role: FOLLOWER
I20240501 20:01:05.278102 27842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47df0b1c907e43b8a88e258dd33b74b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47df0b1c907e43b8a88e258dd33b74b3" member_type: VOTER }
I20240501 20:01:05.278564 27842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47df0b1c907e43b8a88e258dd33b74b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:05.278784 27842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47df0b1c907e43b8a88e258dd33b74b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:05.279014 27842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47df0b1c907e43b8a88e258dd33b74b3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:05.283102 27842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47df0b1c907e43b8a88e258dd33b74b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47df0b1c907e43b8a88e258dd33b74b3" member_type: VOTER }
I20240501 20:01:05.283537 27842 leader_election.cc:302] T 00000000000000000000000000000000 P 47df0b1c907e43b8a88e258dd33b74b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47df0b1c907e43b8a88e258dd33b74b3; no voters:
I20240501 20:01:05.284601 27842 leader_election.cc:288] T 00000000000000000000000000000000 P 47df0b1c907e43b8a88e258dd33b74b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:05.284996 27845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47df0b1c907e43b8a88e258dd33b74b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:05.286306 27845 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47df0b1c907e43b8a88e258dd33b74b3 [term 1 LEADER]: Becoming Leader. State: Replica: 47df0b1c907e43b8a88e258dd33b74b3, State: Running, Role: LEADER
I20240501 20:01:05.286903 27845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47df0b1c907e43b8a88e258dd33b74b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47df0b1c907e43b8a88e258dd33b74b3" member_type: VOTER }
I20240501 20:01:05.287494 27842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47df0b1c907e43b8a88e258dd33b74b3 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:05.290129 27847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47df0b1c907e43b8a88e258dd33b74b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47df0b1c907e43b8a88e258dd33b74b3. Latest consensus state: current_term: 1 leader_uuid: "47df0b1c907e43b8a88e258dd33b74b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47df0b1c907e43b8a88e258dd33b74b3" member_type: VOTER } }
I20240501 20:01:05.290114 27846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47df0b1c907e43b8a88e258dd33b74b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47df0b1c907e43b8a88e258dd33b74b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47df0b1c907e43b8a88e258dd33b74b3" member_type: VOTER } }
I20240501 20:01:05.291132 27846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47df0b1c907e43b8a88e258dd33b74b3 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:05.292176 27847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47df0b1c907e43b8a88e258dd33b74b3 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:05.294703 27852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:05.299209 27852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:05.307755 24854 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 20:01:05.309387 27852 catalog_manager.cc:1347] Generated new cluster ID: f754e60484f04e4eb86dabc2e615b208
I20240501 20:01:05.309669 27852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:05.329901 27852 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:05.331137 27852 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:05.343417 27852 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 47df0b1c907e43b8a88e258dd33b74b3: Generated new TSK 0
I20240501 20:01:05.343989 27852 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:01:05.373904 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:05.379928 27863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:05.381100 27864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:05.382252 27866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:05.382974 24854 server_base.cc:1034] running on GCE node
I20240501 20:01:05.383680 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:01:05.383867 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:05.384017 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593665384002 us; error 0 us; skew 500 ppm
I20240501 20:01:05.384560 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:05.386852 24854 webserver.cc:460] Webserver started at http://127.24.69.129:37647/ using document root <none> and password file <none>
I20240501 20:01:05.387279 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:05.387446 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:05.387796 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:05.389415 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593576104309-24854-0/minicluster-data/ts-0-root/instance:
uuid: "c275a117d77e4bf5a537bd13ed29f652"
format_stamp: "Formatted at 2024-05-01 20:01:05 on dist-test-slave-z8x3"
I20240501 20:01:05.396075 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20240501 20:01:05.399331 27871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:05.400132 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240501 20:01:05.400471 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593576104309-24854-0/minicluster-data/ts-0-root
uuid: "c275a117d77e4bf5a537bd13ed29f652"
format_stamp: "Formatted at 2024-05-01 20:01:05 on dist-test-slave-z8x3"
I20240501 20:01:05.400758 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593576104309-24854-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593576104309-24854-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593576104309-24854-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:05.416694 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:05.417740 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:05.419062 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:05.421218 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:05.421406 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:05.421612 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:05.421780 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:05.458631 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.129:38981
I20240501 20:01:05.458731 27933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.129:38981 every 8 connection(s)
I20240501 20:01:05.463380 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:05.470541 27938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:05.471333 27939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:05.473455 27941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:05.475046 24854 server_base.cc:1034] running on GCE node
I20240501 20:01:05.475925 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:01:05.476183 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:05.476145 27934 heartbeater.cc:347] Connected to a master server at 127.24.69.190:36331
I20240501 20:01:05.476578 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593665476560 us; error 0 us; skew 500 ppm
I20240501 20:01:05.476737 27934 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:05.477156 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:05.477453 27934 heartbeater.cc:511] Master 127.24.69.190:36331 requested a full tablet report, sending...
I20240501 20:01:05.479496 27807 ts_manager.cc:194] Registered new tserver with Master: c275a117d77e4bf5a537bd13ed29f652 (127.24.69.129:38981)
I20240501 20:01:05.479655 24854 webserver.cc:460] Webserver started at http://127.24.69.130:45005/ using document root <none> and password file <none>
I20240501 20:01:05.480185 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:05.480414 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:05.480652 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:05.481338 27807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39122
I20240501 20:01:05.481820 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593576104309-24854-0/minicluster-data/ts-1-root/instance:
uuid: "8dd5e969914c44a48dbcf4a00f87423d"
format_stamp: "Formatted at 2024-05-01 20:01:05 on dist-test-slave-z8x3"
I20240501 20:01:05.486150 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 20:01:05.489209 27946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:05.489939 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:01:05.490199 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593576104309-24854-0/minicluster-data/ts-1-root
uuid: "8dd5e969914c44a48dbcf4a00f87423d"
format_stamp: "Formatted at 2024-05-01 20:01:05 on dist-test-slave-z8x3"
I20240501 20:01:05.490466 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593576104309-24854-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593576104309-24854-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593576104309-24854-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:05.522727 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:05.523887 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:05.525236 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:05.527348 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:05.527540 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:05.527765 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:05.527916 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:05.564703 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.130:41675
I20240501 20:01:05.564810 28008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.130:41675 every 8 connection(s)
I20240501 20:01:05.569165 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:05.576061 28012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:05.578414 28013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:05.579854 28009 heartbeater.cc:347] Connected to a master server at 127.24.69.190:36331
I20240501 20:01:05.580173 28009 heartbeater.cc:464] Registering TS with master...
W20240501 20:01:05.580229 28015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:05.580953 28009 heartbeater.cc:511] Master 127.24.69.190:36331 requested a full tablet report, sending...
I20240501 20:01:05.581100 24854 server_base.cc:1034] running on GCE node
I20240501 20:01:05.582019 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:01:05.582217 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:05.582377 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593665582360 us; error 0 us; skew 500 ppm
I20240501 20:01:05.583055 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:05.583135 27807 ts_manager.cc:194] Registered new tserver with Master: 8dd5e969914c44a48dbcf4a00f87423d (127.24.69.130:41675)
I20240501 20:01:05.584959 27807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39130
I20240501 20:01:05.585870 24854 webserver.cc:460] Webserver started at http://127.24.69.131:36739/ using document root <none> and password file <none>
I20240501 20:01:05.586423 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:05.586668 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:05.587023 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:05.588223 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593576104309-24854-0/minicluster-data/ts-2-root/instance:
uuid: "58b65399a86c4885af3fa874f033ac02"
format_stamp: "Formatted at 2024-05-01 20:01:05 on dist-test-slave-z8x3"
I20240501 20:01:05.592504 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240501 20:01:05.595450 28020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:05.596146 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:01:05.596431 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593576104309-24854-0/minicluster-data/ts-2-root
uuid: "58b65399a86c4885af3fa874f033ac02"
format_stamp: "Formatted at 2024-05-01 20:01:05 on dist-test-slave-z8x3"
I20240501 20:01:05.596695 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593576104309-24854-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593576104309-24854-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593576104309-24854-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:05.607723 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:05.608773 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:05.610070 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:05.612087 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:05.612294 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:05.612502 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:05.612650 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:05.649175 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.131:40747
I20240501 20:01:05.649278 28082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.131:40747 every 8 connection(s)
I20240501 20:01:05.661803 28083 heartbeater.cc:347] Connected to a master server at 127.24.69.190:36331
I20240501 20:01:05.662155 28083 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:05.662837 28083 heartbeater.cc:511] Master 127.24.69.190:36331 requested a full tablet report, sending...
I20240501 20:01:05.664613 27807 ts_manager.cc:194] Registered new tserver with Master: 58b65399a86c4885af3fa874f033ac02 (127.24.69.131:40747)
I20240501 20:01:05.665572 24854 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013294499s
I20240501 20:01:05.665825 27807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39142
I20240501 20:01:06.483633 27934 heartbeater.cc:503] Master 127.24.69.190:36331 was elected leader, sending a full tablet report...
I20240501 20:01:06.587774 28009 heartbeater.cc:503] Master 127.24.69.190:36331 was elected leader, sending a full tablet report...
I20240501 20:01:06.668689 28083 heartbeater.cc:503] Master 127.24.69.190:36331 was elected leader, sending a full tablet report...
I20240501 20:01:06.711519 24854 test_util.cc:268] Using random seed: -227524736
I20240501 20:01:06.734149 27807 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39160:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 20:01:06.736244 27807 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:01:06.783162 27974 tablet_service.cc:1467] Processing CreateTablet for tablet 1f743863be064e9182135a4f7c7f00ed (DEFAULT_TABLE table=test-workload [id=87a9a7d0af824d7bb9d1b7541c1fa63b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:01:06.784999 27974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f743863be064e9182135a4f7c7f00ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:06.791393 27899 tablet_service.cc:1467] Processing CreateTablet for tablet 1f743863be064e9182135a4f7c7f00ed (DEFAULT_TABLE table=test-workload [id=87a9a7d0af824d7bb9d1b7541c1fa63b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:01:06.792038 28048 tablet_service.cc:1467] Processing CreateTablet for tablet 1f743863be064e9182135a4f7c7f00ed (DEFAULT_TABLE table=test-workload [id=87a9a7d0af824d7bb9d1b7541c1fa63b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:01:06.792814 27899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f743863be064e9182135a4f7c7f00ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:06.793402 28048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f743863be064e9182135a4f7c7f00ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:06.805516 28105 tablet_bootstrap.cc:492] T 1f743863be064e9182135a4f7c7f00ed P 8dd5e969914c44a48dbcf4a00f87423d: Bootstrap starting.
I20240501 20:01:06.807462 28106 tablet_bootstrap.cc:492] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02: Bootstrap starting.
I20240501 20:01:06.811228 28107 tablet_bootstrap.cc:492] T 1f743863be064e9182135a4f7c7f00ed P c275a117d77e4bf5a537bd13ed29f652: Bootstrap starting.
I20240501 20:01:06.812214 28105 tablet_bootstrap.cc:654] T 1f743863be064e9182135a4f7c7f00ed P 8dd5e969914c44a48dbcf4a00f87423d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.814092 28106 tablet_bootstrap.cc:654] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.816723 28107 tablet_bootstrap.cc:654] T 1f743863be064e9182135a4f7c7f00ed P c275a117d77e4bf5a537bd13ed29f652: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.818305 28105 tablet_bootstrap.cc:492] T 1f743863be064e9182135a4f7c7f00ed P 8dd5e969914c44a48dbcf4a00f87423d: No bootstrap required, opened a new log
I20240501 20:01:06.818823 28105 ts_tablet_manager.cc:1367] T 1f743863be064e9182135a4f7c7f00ed P 8dd5e969914c44a48dbcf4a00f87423d: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20240501 20:01:06.820005 28106 tablet_bootstrap.cc:492] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02: No bootstrap required, opened a new log
I20240501 20:01:06.820487 28106 ts_tablet_manager.cc:1367] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20240501 20:01:06.821494 28107 tablet_bootstrap.cc:492] T 1f743863be064e9182135a4f7c7f00ed P c275a117d77e4bf5a537bd13ed29f652: No bootstrap required, opened a new log
I20240501 20:01:06.821856 28107 ts_tablet_manager.cc:1367] T 1f743863be064e9182135a4f7c7f00ed P c275a117d77e4bf5a537bd13ed29f652: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240501 20:01:06.821956 28105 raft_consensus.cc:348] T 1f743863be064e9182135a4f7c7f00ed P 8dd5e969914c44a48dbcf4a00f87423d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd5e969914c44a48dbcf4a00f87423d" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 41675 } } peers { permanent_uuid: "c275a117d77e4bf5a537bd13ed29f652" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 38981 } } peers { permanent_uuid: "58b65399a86c4885af3fa874f033ac02" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 40747 } }
I20240501 20:01:06.823067 28105 raft_consensus.cc:374] T 1f743863be064e9182135a4f7c7f00ed P 8dd5e969914c44a48dbcf4a00f87423d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.823462 28105 raft_consensus.cc:729] T 1f743863be064e9182135a4f7c7f00ed P 8dd5e969914c44a48dbcf4a00f87423d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dd5e969914c44a48dbcf4a00f87423d, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.823283 28106 raft_consensus.cc:348] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd5e969914c44a48dbcf4a00f87423d" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 41675 } } peers { permanent_uuid: "c275a117d77e4bf5a537bd13ed29f652" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 38981 } } peers { permanent_uuid: "58b65399a86c4885af3fa874f033ac02" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 40747 } }
I20240501 20:01:06.824272 28106 raft_consensus.cc:374] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.824167 28107 raft_consensus.cc:348] T 1f743863be064e9182135a4f7c7f00ed P c275a117d77e4bf5a537bd13ed29f652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd5e969914c44a48dbcf4a00f87423d" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 41675 } } peers { permanent_uuid: "c275a117d77e4bf5a537bd13ed29f652" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 38981 } } peers { permanent_uuid: "58b65399a86c4885af3fa874f033ac02" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 40747 } }
I20240501 20:01:06.824842 28106 raft_consensus.cc:729] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58b65399a86c4885af3fa874f033ac02, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.825130 28107 raft_consensus.cc:374] T 1f743863be064e9182135a4f7c7f00ed P c275a117d77e4bf5a537bd13ed29f652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.824374 28105 consensus_queue.cc:260] T 1f743863be064e9182135a4f7c7f00ed P 8dd5e969914c44a48dbcf4a00f87423d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd5e969914c44a48dbcf4a00f87423d" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 41675 } } peers { permanent_uuid: "c275a117d77e4bf5a537bd13ed29f652" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 38981 } } peers { permanent_uuid: "58b65399a86c4885af3fa874f033ac02" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 40747 } }
I20240501 20:01:06.825517 28107 raft_consensus.cc:729] T 1f743863be064e9182135a4f7c7f00ed P c275a117d77e4bf5a537bd13ed29f652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c275a117d77e4bf5a537bd13ed29f652, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.825894 28106 consensus_queue.cc:260] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd5e969914c44a48dbcf4a00f87423d" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 41675 } } peers { permanent_uuid: "c275a117d77e4bf5a537bd13ed29f652" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 38981 } } peers { permanent_uuid: "58b65399a86c4885af3fa874f033ac02" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 40747 } }
I20240501 20:01:06.826252 28107 consensus_queue.cc:260] T 1f743863be064e9182135a4f7c7f00ed P c275a117d77e4bf5a537bd13ed29f652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd5e969914c44a48dbcf4a00f87423d" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 41675 } } peers { permanent_uuid: "c275a117d77e4bf5a537bd13ed29f652" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 38981 } } peers { permanent_uuid: "58b65399a86c4885af3fa874f033ac02" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 40747 } }
I20240501 20:01:06.827865 28105 ts_tablet_manager.cc:1398] T 1f743863be064e9182135a4f7c7f00ed P 8dd5e969914c44a48dbcf4a00f87423d: Time spent starting tablet: real 0.009s user 0.005s sys 0.005s
I20240501 20:01:06.829224 28107 ts_tablet_manager.cc:1398] T 1f743863be064e9182135a4f7c7f00ed P c275a117d77e4bf5a537bd13ed29f652: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20240501 20:01:06.831461 28106 ts_tablet_manager.cc:1398] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20240501 20:01:06.872243 28113 raft_consensus.cc:482] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:06.872761 28113 raft_consensus.cc:504] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd5e969914c44a48dbcf4a00f87423d" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 41675 } } peers { permanent_uuid: "c275a117d77e4bf5a537bd13ed29f652" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 38981 } } peers { permanent_uuid: "58b65399a86c4885af3fa874f033ac02" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 40747 } }
I20240501 20:01:06.874918 28113 leader_election.cc:288] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dd5e969914c44a48dbcf4a00f87423d (127.24.69.130:41675), c275a117d77e4bf5a537bd13ed29f652 (127.24.69.129:38981)
I20240501 20:01:06.885571 27984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f743863be064e9182135a4f7c7f00ed" candidate_uuid: "58b65399a86c4885af3fa874f033ac02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dd5e969914c44a48dbcf4a00f87423d" is_pre_election: true
I20240501 20:01:06.886479 27909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f743863be064e9182135a4f7c7f00ed" candidate_uuid: "58b65399a86c4885af3fa874f033ac02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c275a117d77e4bf5a537bd13ed29f652" is_pre_election: true
I20240501 20:01:06.886610 27984 raft_consensus.cc:2454] T 1f743863be064e9182135a4f7c7f00ed P 8dd5e969914c44a48dbcf4a00f87423d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58b65399a86c4885af3fa874f033ac02 in term 0.
I20240501 20:01:06.887084 27909 raft_consensus.cc:2454] T 1f743863be064e9182135a4f7c7f00ed P c275a117d77e4bf5a537bd13ed29f652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58b65399a86c4885af3fa874f033ac02 in term 0.
I20240501 20:01:06.887840 28021 leader_election.cc:302] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58b65399a86c4885af3fa874f033ac02, 8dd5e969914c44a48dbcf4a00f87423d; no voters:
I20240501 20:01:06.888561 28113 raft_consensus.cc:2781] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:06.888847 28113 raft_consensus.cc:482] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:06.889097 28113 raft_consensus.cc:3037] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.895164 28113 raft_consensus.cc:504] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd5e969914c44a48dbcf4a00f87423d" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 41675 } } peers { permanent_uuid: "c275a117d77e4bf5a537bd13ed29f652" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 38981 } } peers { permanent_uuid: "58b65399a86c4885af3fa874f033ac02" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 40747 } }
I20240501 20:01:06.896647 28113 leader_election.cc:288] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02 [CANDIDATE]: Term 1 election: Requested vote from peers 8dd5e969914c44a48dbcf4a00f87423d (127.24.69.130:41675), c275a117d77e4bf5a537bd13ed29f652 (127.24.69.129:38981)
I20240501 20:01:06.897492 27984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f743863be064e9182135a4f7c7f00ed" candidate_uuid: "58b65399a86c4885af3fa874f033ac02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dd5e969914c44a48dbcf4a00f87423d"
I20240501 20:01:06.897727 27909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f743863be064e9182135a4f7c7f00ed" candidate_uuid: "58b65399a86c4885af3fa874f033ac02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c275a117d77e4bf5a537bd13ed29f652"
I20240501 20:01:06.897971 27984 raft_consensus.cc:3037] T 1f743863be064e9182135a4f7c7f00ed P 8dd5e969914c44a48dbcf4a00f87423d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.898277 27909 raft_consensus.cc:3037] T 1f743863be064e9182135a4f7c7f00ed P c275a117d77e4bf5a537bd13ed29f652 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.903707 27984 raft_consensus.cc:2454] T 1f743863be064e9182135a4f7c7f00ed P 8dd5e969914c44a48dbcf4a00f87423d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58b65399a86c4885af3fa874f033ac02 in term 1.
I20240501 20:01:06.904281 27909 raft_consensus.cc:2454] T 1f743863be064e9182135a4f7c7f00ed P c275a117d77e4bf5a537bd13ed29f652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58b65399a86c4885af3fa874f033ac02 in term 1.
I20240501 20:01:06.904753 28021 leader_election.cc:302] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58b65399a86c4885af3fa874f033ac02, 8dd5e969914c44a48dbcf4a00f87423d; no voters:
I20240501 20:01:06.905488 28113 raft_consensus.cc:2781] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:06.906464 28113 raft_consensus.cc:686] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02 [term 1 LEADER]: Becoming Leader. State: Replica: 58b65399a86c4885af3fa874f033ac02, State: Running, Role: LEADER
I20240501 20:01:06.907140 28113 consensus_queue.cc:237] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd5e969914c44a48dbcf4a00f87423d" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 41675 } } peers { permanent_uuid: "c275a117d77e4bf5a537bd13ed29f652" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 38981 } } peers { permanent_uuid: "58b65399a86c4885af3fa874f033ac02" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 40747 } }
I20240501 20:01:06.913268 27807 catalog_manager.cc:5484] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58b65399a86c4885af3fa874f033ac02 (127.24.69.131). New cstate: current_term: 1 leader_uuid: "58b65399a86c4885af3fa874f033ac02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd5e969914c44a48dbcf4a00f87423d" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 41675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c275a117d77e4bf5a537bd13ed29f652" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 38981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58b65399a86c4885af3fa874f033ac02" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 40747 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:07.304402 27855 placement_policy_util.cc:407] tablet 1f743863be064e9182135a4f7c7f00ed: detected majority of replicas (2 of 3) at location L0
I20240501 20:01:07.307651 27860 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:07.308951 27860 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:01:07.309288 27860 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:01:07.346426 28113 consensus_queue.cc:1035] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c275a117d77e4bf5a537bd13ed29f652" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 38981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:07.357106 28113 consensus_queue.cc:1035] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dd5e969914c44a48dbcf4a00f87423d" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 41675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:08.382630 27855 placement_policy_util.cc:407] tablet 1f743863be064e9182135a4f7c7f00ed: detected majority of replicas (2 of 3) at location L0
W20240501 20:01:09.295467 27838 debug-util.cc:398] Leaking SignalData structure 0x7b080004bda0 after lost signal to thread 24857
W20240501 20:01:09.296429 27838 debug-util.cc:398] Leaking SignalData structure 0x7b080015d8e0 after lost signal to thread 27841
W20240501 20:01:09.297362 27838 debug-util.cc:398] Leaking SignalData structure 0x7b0800007720 after lost signal to thread 27933
W20240501 20:01:09.298228 27838 debug-util.cc:398] Leaking SignalData structure 0x7b080008b0c0 after lost signal to thread 28008
W20240501 20:01:09.298991 27838 debug-util.cc:398] Leaking SignalData structure 0x7b08001ac6a0 after lost signal to thread 28082
I20240501 20:01:09.310143 27860 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:09.311410 27860 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:01:09.311697 27860 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:01:09.392369 27855 placement_policy_util.cc:407] tablet 1f743863be064e9182135a4f7c7f00ed: detected majority of replicas (2 of 3) at location L0
I20240501 20:01:10.404953 27855 placement_policy_util.cc:407] tablet 1f743863be064e9182135a4f7c7f00ed: detected majority of replicas (2 of 3) at location L0
I20240501 20:01:11.312578 27860 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:11.313969 27860 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:01:11.314239 27860 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:01:11.412628 27855 placement_policy_util.cc:407] tablet 1f743863be064e9182135a4f7c7f00ed: detected majority of replicas (2 of 3) at location L0
I20240501 20:01:12.419939 27855 placement_policy_util.cc:407] tablet 1f743863be064e9182135a4f7c7f00ed: detected majority of replicas (2 of 3) at location L0
I20240501 20:01:13.315073 27860 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:13.316334 27860 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:01:13.316603 27860 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:01:13.427282 27855 placement_policy_util.cc:407] tablet 1f743863be064e9182135a4f7c7f00ed: detected majority of replicas (2 of 3) at location L0
I20240501 20:01:14.434890 27855 placement_policy_util.cc:407] tablet 1f743863be064e9182135a4f7c7f00ed: detected majority of replicas (2 of 3) at location L0
I20240501 20:01:15.317428 27860 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:15.318688 27860 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:01:15.318989 27860 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:01:15.442168 27855 placement_policy_util.cc:407] tablet 1f743863be064e9182135a4f7c7f00ed: detected majority of replicas (2 of 3) at location L0
I20240501 20:01:16.448415 27855 placement_policy_util.cc:407] tablet 1f743863be064e9182135a4f7c7f00ed: detected majority of replicas (2 of 3) at location L0
I20240501 20:01:17.320044 27860 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:17.321659 27860 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:01:17.321979 27860 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:01:17.454712 27855 placement_policy_util.cc:407] tablet 1f743863be064e9182135a4f7c7f00ed: detected majority of replicas (2 of 3) at location L0
I20240501 20:01:18.008101 24854 tablet_server.cc:178] TabletServer@127.24.69.129:0 shutting down...
I20240501 20:01:18.033555 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:01:18.034297 24854 tablet_replica.cc:330] T 1f743863be064e9182135a4f7c7f00ed P c275a117d77e4bf5a537bd13ed29f652: stopping tablet replica
I20240501 20:01:18.034914 24854 raft_consensus.cc:2229] T 1f743863be064e9182135a4f7c7f00ed P c275a117d77e4bf5a537bd13ed29f652 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:18.035475 24854 raft_consensus.cc:2258] T 1f743863be064e9182135a4f7c7f00ed P c275a117d77e4bf5a537bd13ed29f652 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:18.055004 24854 tablet_server.cc:194] TabletServer@127.24.69.129:0 shutdown complete.
I20240501 20:01:18.066339 24854 tablet_server.cc:178] TabletServer@127.24.69.130:0 shutting down...
I20240501 20:01:18.088732 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:01:18.089430 24854 tablet_replica.cc:330] T 1f743863be064e9182135a4f7c7f00ed P 8dd5e969914c44a48dbcf4a00f87423d: stopping tablet replica
I20240501 20:01:18.089972 24854 raft_consensus.cc:2229] T 1f743863be064e9182135a4f7c7f00ed P 8dd5e969914c44a48dbcf4a00f87423d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:18.090467 24854 raft_consensus.cc:2258] T 1f743863be064e9182135a4f7c7f00ed P 8dd5e969914c44a48dbcf4a00f87423d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:18.110180 24854 tablet_server.cc:194] TabletServer@127.24.69.130:0 shutdown complete.
I20240501 20:01:18.120314 24854 tablet_server.cc:178] TabletServer@127.24.69.131:0 shutting down...
I20240501 20:01:18.142374 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:01:18.143007 24854 tablet_replica.cc:330] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02: stopping tablet replica
I20240501 20:01:18.143594 24854 raft_consensus.cc:2229] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:01:18.144551 24854 raft_consensus.cc:2258] T 1f743863be064e9182135a4f7c7f00ed P 58b65399a86c4885af3fa874f033ac02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:18.162796 24854 tablet_server.cc:194] TabletServer@127.24.69.131:0 shutdown complete.
I20240501 20:01:18.172705 24854 master.cc:537] Master@127.24.69.190:36331 shutting down...
I20240501 20:01:18.191047 24854 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 47df0b1c907e43b8a88e258dd33b74b3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:01:18.191634 24854 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 47df0b1c907e43b8a88e258dd33b74b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:18.191964 24854 tablet_replica.cc:330] T 00000000000000000000000000000000 P 47df0b1c907e43b8a88e258dd33b74b3: stopping tablet replica
I20240501 20:01:18.226418 24854 master.cc:552] Master@127.24.69.190:36331 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (13087 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer-test.cc:516: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] AutoRebalancerTest.TestMaxMovesPerServer (6 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingUnstableCluster
I20240501 20:01:18.266980 24854 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.69.190:34921
I20240501 20:01:18.268040 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:18.274403 28128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:18.275321 28131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:18.276034 28129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:18.276664 24854 server_base.cc:1034] running on GCE node
I20240501 20:01:18.277437 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:01:18.277614 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:18.277724 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593678277713 us; error 0 us; skew 500 ppm
I20240501 20:01:18.278128 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:18.280501 24854 webserver.cc:460] Webserver started at http://127.24.69.190:32785/ using document root <none> and password file <none>
I20240501 20:01:18.280917 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:18.281064 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:18.281288 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:18.282416 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/master-0-root/instance:
uuid: "fb27659c76a544b0a89c0f05bf499012"
format_stamp: "Formatted at 2024-05-01 20:01:18 on dist-test-slave-z8x3"
I20240501 20:01:18.286675 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20240501 20:01:18.289781 28136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:18.290581 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:01:18.290814 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/master-0-root
uuid: "fb27659c76a544b0a89c0f05bf499012"
format_stamp: "Formatted at 2024-05-01 20:01:18 on dist-test-slave-z8x3"
I20240501 20:01:18.291040 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:18.314929 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:18.316022 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:18.351840 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.190:34921
I20240501 20:01:18.351934 28187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.190:34921 every 8 connection(s)
I20240501 20:01:18.355798 28188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:18.366358 28188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb27659c76a544b0a89c0f05bf499012: Bootstrap starting.
I20240501 20:01:18.370424 28188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb27659c76a544b0a89c0f05bf499012: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:18.374181 28188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb27659c76a544b0a89c0f05bf499012: No bootstrap required, opened a new log
I20240501 20:01:18.376004 28188 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb27659c76a544b0a89c0f05bf499012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb27659c76a544b0a89c0f05bf499012" member_type: VOTER }
I20240501 20:01:18.376492 28188 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb27659c76a544b0a89c0f05bf499012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:18.376677 28188 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb27659c76a544b0a89c0f05bf499012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb27659c76a544b0a89c0f05bf499012, State: Initialized, Role: FOLLOWER
I20240501 20:01:18.377144 28188 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb27659c76a544b0a89c0f05bf499012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb27659c76a544b0a89c0f05bf499012" member_type: VOTER }
I20240501 20:01:18.377569 28188 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb27659c76a544b0a89c0f05bf499012 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:18.377753 28188 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb27659c76a544b0a89c0f05bf499012 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:18.377967 28188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb27659c76a544b0a89c0f05bf499012 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:18.381966 28188 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb27659c76a544b0a89c0f05bf499012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb27659c76a544b0a89c0f05bf499012" member_type: VOTER }
I20240501 20:01:18.382401 28188 leader_election.cc:302] T 00000000000000000000000000000000 P fb27659c76a544b0a89c0f05bf499012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb27659c76a544b0a89c0f05bf499012; no voters:
I20240501 20:01:18.383415 28188 leader_election.cc:288] T 00000000000000000000000000000000 P fb27659c76a544b0a89c0f05bf499012 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:18.383780 28191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb27659c76a544b0a89c0f05bf499012 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:18.385073 28191 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb27659c76a544b0a89c0f05bf499012 [term 1 LEADER]: Becoming Leader. State: Replica: fb27659c76a544b0a89c0f05bf499012, State: Running, Role: LEADER
I20240501 20:01:18.385758 28191 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb27659c76a544b0a89c0f05bf499012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb27659c76a544b0a89c0f05bf499012" member_type: VOTER }
I20240501 20:01:18.386319 28188 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb27659c76a544b0a89c0f05bf499012 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:18.391351 28193 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb27659c76a544b0a89c0f05bf499012 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb27659c76a544b0a89c0f05bf499012. Latest consensus state: current_term: 1 leader_uuid: "fb27659c76a544b0a89c0f05bf499012" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb27659c76a544b0a89c0f05bf499012" member_type: VOTER } }
I20240501 20:01:18.391633 28192 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb27659c76a544b0a89c0f05bf499012 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb27659c76a544b0a89c0f05bf499012" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb27659c76a544b0a89c0f05bf499012" member_type: VOTER } }
I20240501 20:01:18.392067 28193 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb27659c76a544b0a89c0f05bf499012 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:18.392414 28192 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb27659c76a544b0a89c0f05bf499012 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:18.393986 28199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:18.398564 28199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:18.401428 24854 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 20:01:18.406760 28199 catalog_manager.cc:1347] Generated new cluster ID: d7c213bf18a04820baaac407e0ef3156
I20240501 20:01:18.406973 28199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:18.420024 28199 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:18.421192 28199 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:18.441093 28199 catalog_manager.cc:5857] T 00000000000000000000000000000000 P fb27659c76a544b0a89c0f05bf499012: Generated new TSK 0
I20240501 20:01:18.441620 28199 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:01:18.468313 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:18.474402 28209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:18.475415 28210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:18.477066 28212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:18.477779 24854 server_base.cc:1034] running on GCE node
I20240501 20:01:18.478488 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:01:18.478680 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:18.478811 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593678478799 us; error 0 us; skew 500 ppm
I20240501 20:01:18.479300 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:18.481576 24854 webserver.cc:460] Webserver started at http://127.24.69.129:37191/ using document root <none> and password file <none>
I20240501 20:01:18.482004 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:18.482157 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:18.482359 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:18.483441 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/ts-0-root/instance:
uuid: "c5e21c75294b4738be7f6c026cb25341"
format_stamp: "Formatted at 2024-05-01 20:01:18 on dist-test-slave-z8x3"
I20240501 20:01:18.487839 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20240501 20:01:18.490869 28217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:18.491803 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:01:18.492079 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/ts-0-root
uuid: "c5e21c75294b4738be7f6c026cb25341"
format_stamp: "Formatted at 2024-05-01 20:01:18 on dist-test-slave-z8x3"
I20240501 20:01:18.492374 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:18.504969 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:18.505983 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:18.507324 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:18.509464 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:18.509660 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:18.509871 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:18.510030 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:18.547273 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.129:37477
I20240501 20:01:18.547361 28279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.129:37477 every 8 connection(s)
I20240501 20:01:18.551723 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:18.560118 28284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:18.561923 28285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:18.565805 28280 heartbeater.cc:347] Connected to a master server at 127.24.69.190:34921
I20240501 20:01:18.566165 28280 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:18.566387 24854 server_base.cc:1034] running on GCE node
I20240501 20:01:18.567041 28280 heartbeater.cc:511] Master 127.24.69.190:34921 requested a full tablet report, sending...
W20240501 20:01:18.567271 28287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:18.568112 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:01:18.568356 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:18.568513 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593678568495 us; error 0 us; skew 500 ppm
I20240501 20:01:18.569139 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:18.569198 28153 ts_manager.cc:194] Registered new tserver with Master: c5e21c75294b4738be7f6c026cb25341 (127.24.69.129:37477)
I20240501 20:01:18.571316 28153 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35362
I20240501 20:01:18.572221 24854 webserver.cc:460] Webserver started at http://127.24.69.130:40231/ using document root <none> and password file <none>
I20240501 20:01:18.572861 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:18.573102 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:18.573423 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:18.574602 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/ts-1-root/instance:
uuid: "9e6105cadd8345f69404f840588e9405"
format_stamp: "Formatted at 2024-05-01 20:01:18 on dist-test-slave-z8x3"
I20240501 20:01:18.578730 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20240501 20:01:18.581881 28292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:18.582588 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:01:18.582824 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/ts-1-root
uuid: "9e6105cadd8345f69404f840588e9405"
format_stamp: "Formatted at 2024-05-01 20:01:18 on dist-test-slave-z8x3"
I20240501 20:01:18.583060 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:18.599879 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:18.601169 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:18.602495 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:18.604636 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:18.604825 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:18.605047 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:18.605195 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:18.641438 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.130:44491
I20240501 20:01:18.641536 28354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.130:44491 every 8 connection(s)
I20240501 20:01:18.646499 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:18.653723 28358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:18.656127 28359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:18.656616 28361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:18.660146 28355 heartbeater.cc:347] Connected to a master server at 127.24.69.190:34921
I20240501 20:01:18.660537 28355 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:18.660740 24854 server_base.cc:1034] running on GCE node
I20240501 20:01:18.661375 28355 heartbeater.cc:511] Master 127.24.69.190:34921 requested a full tablet report, sending...
I20240501 20:01:18.661523 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:01:18.661790 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:18.661970 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593678661953 us; error 0 us; skew 500 ppm
I20240501 20:01:18.662649 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:18.663345 28153 ts_manager.cc:194] Registered new tserver with Master: 9e6105cadd8345f69404f840588e9405 (127.24.69.130:44491)
I20240501 20:01:18.664744 28153 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35366
I20240501 20:01:18.665432 24854 webserver.cc:460] Webserver started at http://127.24.69.131:41141/ using document root <none> and password file <none>
I20240501 20:01:18.665999 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:18.666170 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:18.666416 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:18.667526 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/ts-2-root/instance:
uuid: "fb0239de896a4dfeaf520198010246fb"
format_stamp: "Formatted at 2024-05-01 20:01:18 on dist-test-slave-z8x3"
I20240501 20:01:18.671684 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 20:01:18.674700 28366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:18.675379 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:01:18.675638 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/ts-2-root
uuid: "fb0239de896a4dfeaf520198010246fb"
format_stamp: "Formatted at 2024-05-01 20:01:18 on dist-test-slave-z8x3"
I20240501 20:01:18.675891 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:18.691427 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:18.692559 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:18.693954 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:18.696081 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:18.696316 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:18.696529 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:18.696679 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:18.732800 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.131:35283
I20240501 20:01:18.732905 28428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.131:35283 every 8 connection(s)
I20240501 20:01:18.745374 28429 heartbeater.cc:347] Connected to a master server at 127.24.69.190:34921
I20240501 20:01:18.745726 28429 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:18.746390 28429 heartbeater.cc:511] Master 127.24.69.190:34921 requested a full tablet report, sending...
I20240501 20:01:18.748142 28153 ts_manager.cc:194] Registered new tserver with Master: fb0239de896a4dfeaf520198010246fb (127.24.69.131:35283)
I20240501 20:01:18.748950 24854 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013137462s
I20240501 20:01:18.749476 28153 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35380
I20240501 20:01:19.574007 28280 heartbeater.cc:503] Master 127.24.69.190:34921 was elected leader, sending a full tablet report...
I20240501 20:01:19.667059 28355 heartbeater.cc:503] Master 127.24.69.190:34921 was elected leader, sending a full tablet report...
I20240501 20:01:19.751912 28429 heartbeater.cc:503] Master 127.24.69.190:34921 was elected leader, sending a full tablet report...
I20240501 20:01:19.780809 24854 test_util.cc:268] Using random seed: -214455372
I20240501 20:01:19.803098 28153 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35394:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 20:01:19.805348 28153 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:01:19.859828 28245 tablet_service.cc:1467] Processing CreateTablet for tablet 20cb7f9edf2145daabf966b08220b264 (DEFAULT_TABLE table=test-workload [id=72bed35560fc44488467649b554bc037]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240501 20:01:19.860591 28392 tablet_service.cc:1467] Processing CreateTablet for tablet 268d32d69f8d4591b72359673a8d9b92 (DEFAULT_TABLE table=test-workload [id=72bed35560fc44488467649b554bc037]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240501 20:01:19.861290 28245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20cb7f9edf2145daabf966b08220b264. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:19.861955 28392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 268d32d69f8d4591b72359673a8d9b92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:19.861629 28393 tablet_service.cc:1467] Processing CreateTablet for tablet bd534f7d16d44a478ec5f954776b93aa (DEFAULT_TABLE table=test-workload [id=72bed35560fc44488467649b554bc037]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240501 20:01:19.862282 28320 tablet_service.cc:1467] Processing CreateTablet for tablet 20cb7f9edf2145daabf966b08220b264 (DEFAULT_TABLE table=test-workload [id=72bed35560fc44488467649b554bc037]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240501 20:01:19.862936 28393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd534f7d16d44a478ec5f954776b93aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:19.863554 28320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20cb7f9edf2145daabf966b08220b264. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:19.860591 28394 tablet_service.cc:1467] Processing CreateTablet for tablet 20cb7f9edf2145daabf966b08220b264 (DEFAULT_TABLE table=test-workload [id=72bed35560fc44488467649b554bc037]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240501 20:01:19.865235 28394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20cb7f9edf2145daabf966b08220b264. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:19.865468 28244 tablet_service.cc:1467] Processing CreateTablet for tablet bd534f7d16d44a478ec5f954776b93aa (DEFAULT_TABLE table=test-workload [id=72bed35560fc44488467649b554bc037]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240501 20:01:19.867875 28243 tablet_service.cc:1467] Processing CreateTablet for tablet 268d32d69f8d4591b72359673a8d9b92 (DEFAULT_TABLE table=test-workload [id=72bed35560fc44488467649b554bc037]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240501 20:01:19.869464 28244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd534f7d16d44a478ec5f954776b93aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:19.869354 28318 tablet_service.cc:1467] Processing CreateTablet for tablet 268d32d69f8d4591b72359673a8d9b92 (DEFAULT_TABLE table=test-workload [id=72bed35560fc44488467649b554bc037]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240501 20:01:19.870558 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 268d32d69f8d4591b72359673a8d9b92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:19.872660 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 268d32d69f8d4591b72359673a8d9b92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:19.872983 28319 tablet_service.cc:1467] Processing CreateTablet for tablet bd534f7d16d44a478ec5f954776b93aa (DEFAULT_TABLE table=test-workload [id=72bed35560fc44488467649b554bc037]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240501 20:01:19.874217 28319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd534f7d16d44a478ec5f954776b93aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:19.917343 28450 tablet_bootstrap.cc:492] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341: Bootstrap starting.
I20240501 20:01:19.917737 28449 tablet_bootstrap.cc:492] T 268d32d69f8d4591b72359673a8d9b92 P fb0239de896a4dfeaf520198010246fb: Bootstrap starting.
I20240501 20:01:19.919878 28451 tablet_bootstrap.cc:492] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405: Bootstrap starting.
I20240501 20:01:19.922024 28449 tablet_bootstrap.cc:654] T 268d32d69f8d4591b72359673a8d9b92 P fb0239de896a4dfeaf520198010246fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:19.923979 28450 tablet_bootstrap.cc:654] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:19.926986 28451 tablet_bootstrap.cc:654] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:19.928625 28449 tablet_bootstrap.cc:492] T 268d32d69f8d4591b72359673a8d9b92 P fb0239de896a4dfeaf520198010246fb: No bootstrap required, opened a new log
I20240501 20:01:19.929019 28449 ts_tablet_manager.cc:1367] T 268d32d69f8d4591b72359673a8d9b92 P fb0239de896a4dfeaf520198010246fb: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240501 20:01:19.929824 28450 tablet_bootstrap.cc:492] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341: No bootstrap required, opened a new log
I20240501 20:01:19.930471 28450 ts_tablet_manager.cc:1367] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20240501 20:01:19.931289 28449 raft_consensus.cc:348] T 268d32d69f8d4591b72359673a8d9b92 P fb0239de896a4dfeaf520198010246fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } }
I20240501 20:01:19.932456 28451 tablet_bootstrap.cc:492] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405: No bootstrap required, opened a new log
I20240501 20:01:19.932449 28449 raft_consensus.cc:374] T 268d32d69f8d4591b72359673a8d9b92 P fb0239de896a4dfeaf520198010246fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:19.933065 28449 raft_consensus.cc:729] T 268d32d69f8d4591b72359673a8d9b92 P fb0239de896a4dfeaf520198010246fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0239de896a4dfeaf520198010246fb, State: Initialized, Role: FOLLOWER
I20240501 20:01:19.933166 28451 ts_tablet_manager.cc:1367] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.002s
I20240501 20:01:19.933981 28449 consensus_queue.cc:260] T 268d32d69f8d4591b72359673a8d9b92 P fb0239de896a4dfeaf520198010246fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } }
I20240501 20:01:19.934284 28450 raft_consensus.cc:348] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } }
I20240501 20:01:19.935228 28450 raft_consensus.cc:374] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:19.935619 28450 raft_consensus.cc:729] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e21c75294b4738be7f6c026cb25341, State: Initialized, Role: FOLLOWER
I20240501 20:01:19.936844 28449 ts_tablet_manager.cc:1398] T 268d32d69f8d4591b72359673a8d9b92 P fb0239de896a4dfeaf520198010246fb: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20240501 20:01:19.937374 28450 consensus_queue.cc:260] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } }
I20240501 20:01:19.938364 28449 tablet_bootstrap.cc:492] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb: Bootstrap starting.
I20240501 20:01:19.939190 28451 raft_consensus.cc:348] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } }
I20240501 20:01:19.940207 28451 raft_consensus.cc:374] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:19.940651 28451 raft_consensus.cc:729] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e6105cadd8345f69404f840588e9405, State: Initialized, Role: FOLLOWER
I20240501 20:01:19.942077 28451 consensus_queue.cc:260] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } }
I20240501 20:01:19.945523 28449 tablet_bootstrap.cc:654] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:19.947449 28450 ts_tablet_manager.cc:1398] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341: Time spent starting tablet: real 0.017s user 0.007s sys 0.005s
I20240501 20:01:19.948756 28450 tablet_bootstrap.cc:492] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341: Bootstrap starting.
I20240501 20:01:19.958482 28450 tablet_bootstrap.cc:654] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:19.960304 28449 tablet_bootstrap.cc:492] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb: No bootstrap required, opened a new log
I20240501 20:01:19.960886 28449 ts_tablet_manager.cc:1367] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.006s
I20240501 20:01:19.963294 28449 raft_consensus.cc:348] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } }
I20240501 20:01:19.964108 28449 raft_consensus.cc:374] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:19.964440 28451 ts_tablet_manager.cc:1398] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405: Time spent starting tablet: real 0.031s user 0.008s sys 0.016s
I20240501 20:01:19.964706 28450 tablet_bootstrap.cc:492] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341: No bootstrap required, opened a new log
I20240501 20:01:19.964469 28449 raft_consensus.cc:729] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0239de896a4dfeaf520198010246fb, State: Initialized, Role: FOLLOWER
I20240501 20:01:19.965282 28450 ts_tablet_manager.cc:1367] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.004s
I20240501 20:01:19.965611 28451 tablet_bootstrap.cc:492] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405: Bootstrap starting.
I20240501 20:01:19.965759 28449 consensus_queue.cc:260] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } }
I20240501 20:01:19.968741 28450 raft_consensus.cc:348] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } }
I20240501 20:01:19.969568 28450 raft_consensus.cc:374] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:19.969982 28450 raft_consensus.cc:729] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e21c75294b4738be7f6c026cb25341, State: Initialized, Role: FOLLOWER
I20240501 20:01:19.970850 28450 consensus_queue.cc:260] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } }
I20240501 20:01:19.973110 28451 tablet_bootstrap.cc:654] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:19.973790 28450 ts_tablet_manager.cc:1398] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20240501 20:01:19.974918 28450 tablet_bootstrap.cc:492] T bd534f7d16d44a478ec5f954776b93aa P c5e21c75294b4738be7f6c026cb25341: Bootstrap starting.
I20240501 20:01:19.975189 28456 raft_consensus.cc:482] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:19.975844 28456 raft_consensus.cc:504] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } }
I20240501 20:01:19.978432 28456 leader_election.cc:288] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e6105cadd8345f69404f840588e9405 (127.24.69.130:44491), fb0239de896a4dfeaf520198010246fb (127.24.69.131:35283)
I20240501 20:01:19.985356 28449 ts_tablet_manager.cc:1398] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20240501 20:01:19.986661 28450 tablet_bootstrap.cc:654] T bd534f7d16d44a478ec5f954776b93aa P c5e21c75294b4738be7f6c026cb25341: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:19.987624 28451 tablet_bootstrap.cc:492] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405: No bootstrap required, opened a new log
I20240501 20:01:19.988310 28451 ts_tablet_manager.cc:1367] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.007s
I20240501 20:01:19.988701 28449 tablet_bootstrap.cc:492] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb: Bootstrap starting.
I20240501 20:01:19.991623 28451 raft_consensus.cc:348] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } }
I20240501 20:01:19.998952 28451 raft_consensus.cc:374] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:19.999434 28451 raft_consensus.cc:729] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e6105cadd8345f69404f840588e9405, State: Initialized, Role: FOLLOWER
I20240501 20:01:19.999904 28455 raft_consensus.cc:482] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:20.000437 28455 raft_consensus.cc:504] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } }
I20240501 20:01:20.002709 28449 tablet_bootstrap.cc:654] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:20.006027 28455 leader_election.cc:288] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5e21c75294b4738be7f6c026cb25341 (127.24.69.129:37477), 9e6105cadd8345f69404f840588e9405 (127.24.69.130:44491)
I20240501 20:01:20.003319 28451 consensus_queue.cc:260] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } }
I20240501 20:01:20.009305 28330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "268d32d69f8d4591b72359673a8d9b92" candidate_uuid: "c5e21c75294b4738be7f6c026cb25341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e6105cadd8345f69404f840588e9405" is_pre_election: true
I20240501 20:01:20.010116 28330 raft_consensus.cc:2454] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e21c75294b4738be7f6c026cb25341 in term 0.
I20240501 20:01:20.011569 28219 leader_election.cc:302] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e6105cadd8345f69404f840588e9405, c5e21c75294b4738be7f6c026cb25341; no voters:
I20240501 20:01:20.012418 28456 raft_consensus.cc:2781] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:20.012763 28456 raft_consensus.cc:482] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:20.013073 28456 raft_consensus.cc:3037] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:20.016286 28451 ts_tablet_manager.cc:1398] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405: Time spent starting tablet: real 0.028s user 0.005s sys 0.002s
I20240501 20:01:20.017247 28451 tablet_bootstrap.cc:492] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405: Bootstrap starting.
I20240501 20:01:20.021648 28456 raft_consensus.cc:504] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } }
I20240501 20:01:20.022634 28451 tablet_bootstrap.cc:654] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:20.024564 28330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "268d32d69f8d4591b72359673a8d9b92" candidate_uuid: "c5e21c75294b4738be7f6c026cb25341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e6105cadd8345f69404f840588e9405"
I20240501 20:01:20.025175 28330 raft_consensus.cc:3037] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:20.026741 28404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "268d32d69f8d4591b72359673a8d9b92" candidate_uuid: "c5e21c75294b4738be7f6c026cb25341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb0239de896a4dfeaf520198010246fb" is_pre_election: true
I20240501 20:01:20.027307 28404 raft_consensus.cc:2454] T 268d32d69f8d4591b72359673a8d9b92 P fb0239de896a4dfeaf520198010246fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e21c75294b4738be7f6c026cb25341 in term 0.
I20240501 20:01:20.028587 28449 tablet_bootstrap.cc:492] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb: No bootstrap required, opened a new log
I20240501 20:01:20.029063 28449 ts_tablet_manager.cc:1367] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb: Time spent bootstrapping tablet: real 0.043s user 0.010s sys 0.002s
I20240501 20:01:20.030922 28449 raft_consensus.cc:348] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } }
I20240501 20:01:20.031411 28449 raft_consensus.cc:374] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:20.031733 28449 raft_consensus.cc:729] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0239de896a4dfeaf520198010246fb, State: Initialized, Role: FOLLOWER
I20240501 20:01:20.034544 28404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "268d32d69f8d4591b72359673a8d9b92" candidate_uuid: "c5e21c75294b4738be7f6c026cb25341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb0239de896a4dfeaf520198010246fb"
I20240501 20:01:20.035032 28450 tablet_bootstrap.cc:492] T bd534f7d16d44a478ec5f954776b93aa P c5e21c75294b4738be7f6c026cb25341: No bootstrap required, opened a new log
I20240501 20:01:20.035091 28404 raft_consensus.cc:3037] T 268d32d69f8d4591b72359673a8d9b92 P fb0239de896a4dfeaf520198010246fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:20.035585 28450 ts_tablet_manager.cc:1367] T bd534f7d16d44a478ec5f954776b93aa P c5e21c75294b4738be7f6c026cb25341: Time spent bootstrapping tablet: real 0.061s user 0.018s sys 0.009s
I20240501 20:01:20.038357 28450 raft_consensus.cc:348] T bd534f7d16d44a478ec5f954776b93aa P c5e21c75294b4738be7f6c026cb25341 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } }
I20240501 20:01:20.039031 28450 raft_consensus.cc:374] T bd534f7d16d44a478ec5f954776b93aa P c5e21c75294b4738be7f6c026cb25341 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:20.039297 28450 raft_consensus.cc:729] T bd534f7d16d44a478ec5f954776b93aa P c5e21c75294b4738be7f6c026cb25341 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e21c75294b4738be7f6c026cb25341, State: Initialized, Role: FOLLOWER
I20240501 20:01:20.039968 28450 consensus_queue.cc:260] T bd534f7d16d44a478ec5f954776b93aa P c5e21c75294b4738be7f6c026cb25341 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } }
I20240501 20:01:20.040406 28330 raft_consensus.cc:2454] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e21c75294b4738be7f6c026cb25341 in term 1.
I20240501 20:01:20.032608 28449 consensus_queue.cc:260] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } }
I20240501 20:01:20.045758 28449 ts_tablet_manager.cc:1398] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20240501 20:01:20.042425 28404 raft_consensus.cc:2454] T 268d32d69f8d4591b72359673a8d9b92 P fb0239de896a4dfeaf520198010246fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e21c75294b4738be7f6c026cb25341 in term 1.
I20240501 20:01:20.049186 28220 leader_election.cc:302] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5e21c75294b4738be7f6c026cb25341, fb0239de896a4dfeaf520198010246fb; no voters:
I20240501 20:01:20.053078 28468 raft_consensus.cc:2781] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:20.053720 28468 raft_consensus.cc:686] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [term 1 LEADER]: Becoming Leader. State: Replica: c5e21c75294b4738be7f6c026cb25341, State: Running, Role: LEADER
I20240501 20:01:20.055363 28450 ts_tablet_manager.cc:1398] T bd534f7d16d44a478ec5f954776b93aa P c5e21c75294b4738be7f6c026cb25341: Time spent starting tablet: real 0.019s user 0.004s sys 0.014s
I20240501 20:01:20.054761 28468 consensus_queue.cc:237] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } }
I20240501 20:01:20.042963 28456 leader_election.cc:288] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [CANDIDATE]: Term 1 election: Requested vote from peers 9e6105cadd8345f69404f840588e9405 (127.24.69.130:44491), fb0239de896a4dfeaf520198010246fb (127.24.69.131:35283)
I20240501 20:01:20.071027 28330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd534f7d16d44a478ec5f954776b93aa" candidate_uuid: "fb0239de896a4dfeaf520198010246fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e6105cadd8345f69404f840588e9405" is_pre_election: true
W20240501 20:01:20.072688 28368 leader_election.cc:341] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9e6105cadd8345f69404f840588e9405 (127.24.69.130:44491): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:20.072746 28255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd534f7d16d44a478ec5f954776b93aa" candidate_uuid: "fb0239de896a4dfeaf520198010246fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e21c75294b4738be7f6c026cb25341" is_pre_election: true
I20240501 20:01:20.073390 28255 raft_consensus.cc:2454] T bd534f7d16d44a478ec5f954776b93aa P c5e21c75294b4738be7f6c026cb25341 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb0239de896a4dfeaf520198010246fb in term 0.
I20240501 20:01:20.075551 28451 tablet_bootstrap.cc:492] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405: No bootstrap required, opened a new log
I20240501 20:01:20.075469 28370 leader_election.cc:302] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5e21c75294b4738be7f6c026cb25341, fb0239de896a4dfeaf520198010246fb; no voters: 9e6105cadd8345f69404f840588e9405
I20240501 20:01:20.076093 28451 ts_tablet_manager.cc:1367] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405: Time spent bootstrapping tablet: real 0.059s user 0.021s sys 0.030s
I20240501 20:01:20.076422 28455 raft_consensus.cc:2781] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:20.076826 28455 raft_consensus.cc:482] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:20.077172 28455 raft_consensus.cc:3037] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:20.078380 28451 raft_consensus.cc:348] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } }
I20240501 20:01:20.078909 28451 raft_consensus.cc:374] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:20.079151 28451 raft_consensus.cc:729] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e6105cadd8345f69404f840588e9405, State: Initialized, Role: FOLLOWER
I20240501 20:01:20.078215 28153 catalog_manager.cc:5484] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e21c75294b4738be7f6c026cb25341 (127.24.69.129). New cstate: current_term: 1 leader_uuid: "c5e21c75294b4738be7f6c026cb25341" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:20.079756 28451 consensus_queue.cc:260] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } }
I20240501 20:01:20.081660 28451 ts_tablet_manager.cc:1398] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:01:20.084017 28455 raft_consensus.cc:504] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } }
I20240501 20:01:20.085640 28455 leader_election.cc:288] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [CANDIDATE]: Term 1 election: Requested vote from peers c5e21c75294b4738be7f6c026cb25341 (127.24.69.129:37477), 9e6105cadd8345f69404f840588e9405 (127.24.69.130:44491)
I20240501 20:01:20.086673 28330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd534f7d16d44a478ec5f954776b93aa" candidate_uuid: "fb0239de896a4dfeaf520198010246fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e6105cadd8345f69404f840588e9405"
I20240501 20:01:20.087296 28330 raft_consensus.cc:3037] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:20.091768 28330 raft_consensus.cc:2454] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb0239de896a4dfeaf520198010246fb in term 1.
I20240501 20:01:20.092605 28255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd534f7d16d44a478ec5f954776b93aa" candidate_uuid: "fb0239de896a4dfeaf520198010246fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e21c75294b4738be7f6c026cb25341"
I20240501 20:01:20.093189 28255 raft_consensus.cc:3037] T bd534f7d16d44a478ec5f954776b93aa P c5e21c75294b4738be7f6c026cb25341 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:20.094753 28368 leader_election.cc:302] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e6105cadd8345f69404f840588e9405, fb0239de896a4dfeaf520198010246fb; no voters:
I20240501 20:01:20.095510 28455 raft_consensus.cc:2781] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:20.096853 28455 raft_consensus.cc:686] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [term 1 LEADER]: Becoming Leader. State: Replica: fb0239de896a4dfeaf520198010246fb, State: Running, Role: LEADER
I20240501 20:01:20.098181 28455 consensus_queue.cc:237] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } }
I20240501 20:01:20.099488 28255 raft_consensus.cc:2454] T bd534f7d16d44a478ec5f954776b93aa P c5e21c75294b4738be7f6c026cb25341 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb0239de896a4dfeaf520198010246fb in term 1.
I20240501 20:01:20.105837 28152 catalog_manager.cc:5484] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb reported cstate change: term changed from 0 to 1, leader changed from <none> to fb0239de896a4dfeaf520198010246fb (127.24.69.131). New cstate: current_term: 1 leader_uuid: "fb0239de896a4dfeaf520198010246fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:20.139477 28468 raft_consensus.cc:482] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:20.139863 28468 raft_consensus.cc:504] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } }
I20240501 20:01:20.141469 28468 leader_election.cc:288] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb0239de896a4dfeaf520198010246fb (127.24.69.131:35283), 9e6105cadd8345f69404f840588e9405 (127.24.69.130:44491)
I20240501 20:01:20.142122 28404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cb7f9edf2145daabf966b08220b264" candidate_uuid: "c5e21c75294b4738be7f6c026cb25341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb0239de896a4dfeaf520198010246fb" is_pre_election: true
I20240501 20:01:20.142463 28330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cb7f9edf2145daabf966b08220b264" candidate_uuid: "c5e21c75294b4738be7f6c026cb25341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e6105cadd8345f69404f840588e9405" is_pre_election: true
I20240501 20:01:20.142796 28404 raft_consensus.cc:2454] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e21c75294b4738be7f6c026cb25341 in term 0.
I20240501 20:01:20.143110 28330 raft_consensus.cc:2454] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e21c75294b4738be7f6c026cb25341 in term 0.
I20240501 20:01:20.143784 28220 leader_election.cc:302] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5e21c75294b4738be7f6c026cb25341, fb0239de896a4dfeaf520198010246fb; no voters:
I20240501 20:01:20.144403 28468 raft_consensus.cc:2781] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:20.144738 28468 raft_consensus.cc:482] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:20.144955 28468 raft_consensus.cc:3037] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:20.149380 28468 raft_consensus.cc:504] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } }
I20240501 20:01:20.150816 28468 leader_election.cc:288] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [CANDIDATE]: Term 1 election: Requested vote from peers fb0239de896a4dfeaf520198010246fb (127.24.69.131:35283), 9e6105cadd8345f69404f840588e9405 (127.24.69.130:44491)
I20240501 20:01:20.151495 28404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cb7f9edf2145daabf966b08220b264" candidate_uuid: "c5e21c75294b4738be7f6c026cb25341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb0239de896a4dfeaf520198010246fb"
I20240501 20:01:20.151744 28330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cb7f9edf2145daabf966b08220b264" candidate_uuid: "c5e21c75294b4738be7f6c026cb25341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e6105cadd8345f69404f840588e9405"
I20240501 20:01:20.152081 28404 raft_consensus.cc:3037] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:20.152285 28330 raft_consensus.cc:3037] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:20.158120 28330 raft_consensus.cc:2454] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e21c75294b4738be7f6c026cb25341 in term 1.
I20240501 20:01:20.158120 28404 raft_consensus.cc:2454] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e21c75294b4738be7f6c026cb25341 in term 1.
I20240501 20:01:20.159217 28220 leader_election.cc:302] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5e21c75294b4738be7f6c026cb25341, fb0239de896a4dfeaf520198010246fb; no voters:
I20240501 20:01:20.159812 28468 raft_consensus.cc:2781] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:20.160197 28468 raft_consensus.cc:686] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [term 1 LEADER]: Becoming Leader. State: Replica: c5e21c75294b4738be7f6c026cb25341, State: Running, Role: LEADER
I20240501 20:01:20.160775 28468 consensus_queue.cc:237] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } }
I20240501 20:01:20.167363 28152 catalog_manager.cc:5484] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e21c75294b4738be7f6c026cb25341 (127.24.69.129). New cstate: current_term: 1 leader_uuid: "c5e21c75294b4738be7f6c026cb25341" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:20.232201 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:20.238358 28474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:20.239475 28475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:20.242130 28477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:20.242834 24854 server_base.cc:1034] running on GCE node
I20240501 20:01:20.243937 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:01:20.244131 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:20.244334 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593680244314 us; error 0 us; skew 500 ppm
I20240501 20:01:20.244825 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:20.247130 24854 webserver.cc:460] Webserver started at http://127.24.69.132:46283/ using document root <none> and password file <none>
I20240501 20:01:20.247619 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:20.247786 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:20.247987 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:20.248999 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/ts-3-root/instance:
uuid: "040d1626e04e4dc1b63a334540590bfb"
format_stamp: "Formatted at 2024-05-01 20:01:20 on dist-test-slave-z8x3"
I20240501 20:01:20.253387 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 20:01:20.256335 28482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:20.257046 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:01:20.257298 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/ts-3-root
uuid: "040d1626e04e4dc1b63a334540590bfb"
format_stamp: "Formatted at 2024-05-01 20:01:20 on dist-test-slave-z8x3"
I20240501 20:01:20.257531 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:20.281982 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:20.283190 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:20.284682 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:20.286893 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:20.287091 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:20.287376 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:20.287536 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:20.313539 28468 consensus_queue.cc:1035] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:01:20.321303 28468 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20240501 20:01:20.321591 28468 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20240501 20:01:20.328990 28469 consensus_queue.cc:1035] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:20.340710 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.132:45365
I20240501 20:01:20.340873 28548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.132:45365 every 8 connection(s)
I20240501 20:01:20.355135 28549 heartbeater.cc:347] Connected to a master server at 127.24.69.190:34921
I20240501 20:01:20.355433 28549 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:20.356047 28549 heartbeater.cc:511] Master 127.24.69.190:34921 requested a full tablet report, sending...
I20240501 20:01:20.357606 28152 ts_manager.cc:194] Registered new tserver with Master: 040d1626e04e4dc1b63a334540590bfb (127.24.69.132:45365)
I20240501 20:01:20.359015 28152 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35402
I20240501 20:01:20.401196 28206 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:20.405694 28455 consensus_queue.cc:1035] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:20.424474 28255 consensus_queue.cc:237] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } attrs { replace: true } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: true } }
I20240501 20:01:20.426841 28455 consensus_queue.cc:1035] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:20.441287 28330 raft_consensus.cc:1261] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Refusing update from remote peer c5e21c75294b4738be7f6c026cb25341: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:20.443039 28558 consensus_queue.cc:1035] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:20.443183 28255 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "20cb7f9edf2145daabf966b08220b264"
dest_uuid: "c5e21c75294b4738be7f6c026cb25341"
mode: GRACEFUL
new_leader_uuid: "9e6105cadd8345f69404f840588e9405"
from {username='slave'} at 127.0.0.1:45802
I20240501 20:01:20.444082 28255 raft_consensus.cc:595] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [term 1 LEADER]: Received request to transfer leadership to TS 9e6105cadd8345f69404f840588e9405
I20240501 20:01:20.446269 28206 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 20:01:20.446734 28206 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:01:20.441402 28404 raft_consensus.cc:1261] T 268d32d69f8d4591b72359673a8d9b92 P fb0239de896a4dfeaf520198010246fb [term 1 FOLLOWER]: Refusing update from remote peer c5e21c75294b4738be7f6c026cb25341: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:20.452432 28456 consensus_queue.cc:1035] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:20.453989 28330 raft_consensus.cc:2932] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 040d1626e04e4dc1b63a334540590bfb (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } attrs { replace: true } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: true } } }
W20240501 20:01:20.456445 28221 consensus_peers.cc:485] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 -> Peer 040d1626e04e4dc1b63a334540590bfb (127.24.69.132:45365): Couldn't send request to peer 040d1626e04e4dc1b63a334540590bfb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 268d32d69f8d4591b72359673a8d9b92. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:20.457237 28404 raft_consensus.cc:2932] T 268d32d69f8d4591b72359673a8d9b92 P fb0239de896a4dfeaf520198010246fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 040d1626e04e4dc1b63a334540590bfb (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } attrs { replace: true } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: true } } }
I20240501 20:01:20.466185 28152 catalog_manager.cc:5484] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 reported cstate change: config changed from index -1 to 2, NON_VOTER 040d1626e04e4dc1b63a334540590bfb (127.24.69.132) added. New cstate: current_term: 1 leader_uuid: "c5e21c75294b4738be7f6c026cb25341" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } attrs { replace: true } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: true } } }
I20240501 20:01:20.473503 28456 consensus_queue.cc:1035] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:20.482210 28469 consensus_queue.cc:1035] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:20.672681 28563 ts_tablet_manager.cc:902] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb: Initiating tablet copy from peer c5e21c75294b4738be7f6c026cb25341 (127.24.69.129:37477)
I20240501 20:01:20.675489 28563 tablet_copy_client.cc:308] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb: tablet copy: Beginning tablet copy session from remote peer at address 127.24.69.129:37477
I20240501 20:01:20.683584 28265 tablet_copy_service.cc:140] P c5e21c75294b4738be7f6c026cb25341: Received BeginTabletCopySession request for tablet 268d32d69f8d4591b72359673a8d9b92 from peer 040d1626e04e4dc1b63a334540590bfb ({username='slave'} at 127.0.0.1:45816)
I20240501 20:01:20.684149 28265 tablet_copy_service.cc:161] P c5e21c75294b4738be7f6c026cb25341: Beginning new tablet copy session on tablet 268d32d69f8d4591b72359673a8d9b92 from peer 040d1626e04e4dc1b63a334540590bfb at {username='slave'} at 127.0.0.1:45816: session id = 040d1626e04e4dc1b63a334540590bfb-268d32d69f8d4591b72359673a8d9b92
I20240501 20:01:20.689424 28265 tablet_copy_source_session.cc:205] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:01:20.692114 28563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 268d32d69f8d4591b72359673a8d9b92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:20.702620 28563 tablet_copy_client.cc:783] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb: tablet copy: Starting download of 0 data blocks...
I20240501 20:01:20.703334 28563 tablet_copy_client.cc:647] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:20.706699 28563 tablet_copy_client.cc:523] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:20.711894 28563 tablet_bootstrap.cc:492] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb: Bootstrap starting.
I20240501 20:01:20.719225 28456 raft_consensus.cc:2932] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 040d1626e04e4dc1b63a334540590bfb (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } attrs { replace: true } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: true } } }
I20240501 20:01:20.725996 28563 tablet_bootstrap.cc:492] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:01:20.727038 28563 tablet_bootstrap.cc:492] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb: Bootstrap complete.
I20240501 20:01:20.727679 28563 ts_tablet_manager.cc:1367] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.008s
I20240501 20:01:20.730571 28563 raft_consensus.cc:348] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } }
I20240501 20:01:20.731916 28563 raft_consensus.cc:729] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 040d1626e04e4dc1b63a334540590bfb, State: Initialized, Role: LEARNER
I20240501 20:01:20.732616 28563 consensus_queue.cc:260] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } attrs { replace: true } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: true } }
I20240501 20:01:20.735092 28549 heartbeater.cc:503] Master 127.24.69.190:34921 was elected leader, sending a full tablet report...
I20240501 20:01:20.735611 28563 ts_tablet_manager.cc:1398] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20240501 20:01:20.737450 28265 tablet_copy_service.cc:342] P c5e21c75294b4738be7f6c026cb25341: Request end of tablet copy session 040d1626e04e4dc1b63a334540590bfb-268d32d69f8d4591b72359673a8d9b92 received from {username='slave'} at 127.0.0.1:45816
I20240501 20:01:20.737834 28265 tablet_copy_service.cc:433] P c5e21c75294b4738be7f6c026cb25341: ending tablet copy session 040d1626e04e4dc1b63a334540590bfb-268d32d69f8d4591b72359673a8d9b92 on tablet 268d32d69f8d4591b72359673a8d9b92 with peer 040d1626e04e4dc1b63a334540590bfb
I20240501 20:01:20.744340 28404 consensus_queue.cc:237] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } attrs { replace: true } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: true } }
I20240501 20:01:20.749799 28330 raft_consensus.cc:1261] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Refusing update from remote peer fb0239de896a4dfeaf520198010246fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:20.751495 28568 consensus_queue.cc:1035] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:20.752652 28255 raft_consensus.cc:1261] T bd534f7d16d44a478ec5f954776b93aa P c5e21c75294b4738be7f6c026cb25341 [term 1 FOLLOWER]: Refusing update from remote peer fb0239de896a4dfeaf520198010246fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:20.754101 28568 consensus_queue.cc:1035] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:01:20.761837 28370 consensus_peers.cc:485] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb -> Peer 040d1626e04e4dc1b63a334540590bfb (127.24.69.132:45365): Couldn't send request to peer 040d1626e04e4dc1b63a334540590bfb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd534f7d16d44a478ec5f954776b93aa. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:20.761535 28330 raft_consensus.cc:2932] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 040d1626e04e4dc1b63a334540590bfb (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } attrs { replace: true } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: true } } }
I20240501 20:01:20.762153 28255 raft_consensus.cc:2932] T bd534f7d16d44a478ec5f954776b93aa P c5e21c75294b4738be7f6c026cb25341 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 040d1626e04e4dc1b63a334540590bfb (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } attrs { replace: true } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: true } } }
I20240501 20:01:20.771299 28153 catalog_manager.cc:5484] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405 reported cstate change: config changed from index -1 to 2, NON_VOTER 040d1626e04e4dc1b63a334540590bfb (127.24.69.132) added. New cstate: current_term: 1 leader_uuid: "fb0239de896a4dfeaf520198010246fb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } attrs { replace: true } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: true } } }
I20240501 20:01:20.827581 28558 raft_consensus.cc:979] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341: : Instructing follower 9e6105cadd8345f69404f840588e9405 to start an election
I20240501 20:01:20.828116 28469 raft_consensus.cc:1067] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [term 1 LEADER]: Signalling peer 9e6105cadd8345f69404f840588e9405 to start an election
I20240501 20:01:20.829686 28330 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "20cb7f9edf2145daabf966b08220b264"
dest_uuid: "9e6105cadd8345f69404f840588e9405"
from {username='slave'} at 127.0.0.1:35148
I20240501 20:01:20.830272 28330 raft_consensus.cc:482] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:01:20.830546 28330 raft_consensus.cc:3037] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:20.834956 28330 raft_consensus.cc:504] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } }
I20240501 20:01:20.836642 28330 leader_election.cc:288] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [CANDIDATE]: Term 2 election: Requested vote from peers c5e21c75294b4738be7f6c026cb25341 (127.24.69.129:37477), fb0239de896a4dfeaf520198010246fb (127.24.69.131:35283)
I20240501 20:01:20.846107 28255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cb7f9edf2145daabf966b08220b264" candidate_uuid: "9e6105cadd8345f69404f840588e9405" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c5e21c75294b4738be7f6c026cb25341"
I20240501 20:01:20.846235 28404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cb7f9edf2145daabf966b08220b264" candidate_uuid: "9e6105cadd8345f69404f840588e9405" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fb0239de896a4dfeaf520198010246fb"
I20240501 20:01:20.846632 28255 raft_consensus.cc:3032] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:20.846771 28404 raft_consensus.cc:3037] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:20.846904 28255 raft_consensus.cc:729] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c5e21c75294b4738be7f6c026cb25341, State: Running, Role: LEADER
I20240501 20:01:20.847601 28255 consensus_queue.cc:260] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } }
I20240501 20:01:20.848546 28255 raft_consensus.cc:3037] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:20.851243 28404 raft_consensus.cc:2454] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e6105cadd8345f69404f840588e9405 in term 2.
I20240501 20:01:20.852228 28295 leader_election.cc:302] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e6105cadd8345f69404f840588e9405, fb0239de896a4dfeaf520198010246fb; no voters:
I20240501 20:01:20.853055 28255 raft_consensus.cc:2454] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e6105cadd8345f69404f840588e9405 in term 2.
I20240501 20:01:20.853163 28457 raft_consensus.cc:2781] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:20.854130 28457 raft_consensus.cc:686] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [term 2 LEADER]: Becoming Leader. State: Replica: 9e6105cadd8345f69404f840588e9405, State: Running, Role: LEADER
I20240501 20:01:20.854920 28457 consensus_queue.cc:237] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } }
I20240501 20:01:20.860483 28152 catalog_manager.cc:5484] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 reported cstate change: term changed from 1 to 2, leader changed from c5e21c75294b4738be7f6c026cb25341 (127.24.69.129) to 9e6105cadd8345f69404f840588e9405 (127.24.69.130). New cstate: current_term: 2 leader_uuid: "9e6105cadd8345f69404f840588e9405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:20.921267 28520 raft_consensus.cc:1203] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:01:20.922062 28520 raft_consensus.cc:2932] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [term 1 LEARNER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 040d1626e04e4dc1b63a334540590bfb (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } attrs { replace: true } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: true } } }
I20240501 20:01:20.959076 28471 raft_consensus.cc:2932] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 040d1626e04e4dc1b63a334540590bfb (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } attrs { replace: true } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: true } } }
I20240501 20:01:21.123772 28563 ts_tablet_manager.cc:902] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb: Initiating tablet copy from peer fb0239de896a4dfeaf520198010246fb (127.24.69.131:35283)
I20240501 20:01:21.127351 28563 tablet_copy_client.cc:308] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb: tablet copy: Beginning tablet copy session from remote peer at address 127.24.69.131:35283
I20240501 20:01:21.135658 28414 tablet_copy_service.cc:140] P fb0239de896a4dfeaf520198010246fb: Received BeginTabletCopySession request for tablet bd534f7d16d44a478ec5f954776b93aa from peer 040d1626e04e4dc1b63a334540590bfb ({username='slave'} at 127.0.0.1:58832)
I20240501 20:01:21.136147 28414 tablet_copy_service.cc:161] P fb0239de896a4dfeaf520198010246fb: Beginning new tablet copy session on tablet bd534f7d16d44a478ec5f954776b93aa from peer 040d1626e04e4dc1b63a334540590bfb at {username='slave'} at 127.0.0.1:58832: session id = 040d1626e04e4dc1b63a334540590bfb-bd534f7d16d44a478ec5f954776b93aa
I20240501 20:01:21.141348 28414 tablet_copy_source_session.cc:205] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:01:21.143611 28563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd534f7d16d44a478ec5f954776b93aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:21.154611 28563 tablet_copy_client.cc:783] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb: tablet copy: Starting download of 0 data blocks...
I20240501 20:01:21.155100 28563 tablet_copy_client.cc:647] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:21.158308 28563 tablet_copy_client.cc:523] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:21.165509 28563 tablet_bootstrap.cc:492] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb: Bootstrap starting.
I20240501 20:01:21.182360 28563 tablet_bootstrap.cc:492] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:01:21.183048 28563 tablet_bootstrap.cc:492] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb: Bootstrap complete.
I20240501 20:01:21.183553 28563 ts_tablet_manager.cc:1367] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.004s
I20240501 20:01:21.185772 28563 raft_consensus.cc:348] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } attrs { replace: true } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: true } }
I20240501 20:01:21.186388 28563 raft_consensus.cc:729] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 040d1626e04e4dc1b63a334540590bfb, State: Initialized, Role: LEARNER
I20240501 20:01:21.186993 28563 consensus_queue.cc:260] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } attrs { replace: true } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: true } }
I20240501 20:01:21.189036 28563 ts_tablet_manager.cc:1398] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20240501 20:01:21.190480 28414 tablet_copy_service.cc:342] P fb0239de896a4dfeaf520198010246fb: Request end of tablet copy session 040d1626e04e4dc1b63a334540590bfb-bd534f7d16d44a478ec5f954776b93aa received from {username='slave'} at 127.0.0.1:58832
I20240501 20:01:21.190888 28414 tablet_copy_service.cc:433] P fb0239de896a4dfeaf520198010246fb: ending tablet copy session 040d1626e04e4dc1b63a334540590bfb-bd534f7d16d44a478ec5f954776b93aa on tablet bd534f7d16d44a478ec5f954776b93aa with peer 040d1626e04e4dc1b63a334540590bfb
I20240501 20:01:21.201062 28404 raft_consensus.cc:1261] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [term 2 FOLLOWER]: Refusing update from remote peer 9e6105cadd8345f69404f840588e9405: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:21.202227 28457 consensus_queue.cc:1035] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:21.205516 28255 raft_consensus.cc:1261] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [term 2 FOLLOWER]: Refusing update from remote peer 9e6105cadd8345f69404f840588e9405: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:21.209071 28584 consensus_queue.cc:1035] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:21.370085 28520 raft_consensus.cc:1203] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:01:21.425925 28456 raft_consensus.cc:1050] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341: attempting to promote NON_VOTER 040d1626e04e4dc1b63a334540590bfb to VOTER
I20240501 20:01:21.427811 28456 consensus_queue.cc:237] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } attrs { replace: true } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } }
I20240501 20:01:21.432699 28403 raft_consensus.cc:1261] T 268d32d69f8d4591b72359673a8d9b92 P fb0239de896a4dfeaf520198010246fb [term 1 FOLLOWER]: Refusing update from remote peer c5e21c75294b4738be7f6c026cb25341: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:21.432715 28520 raft_consensus.cc:1261] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [term 1 LEARNER]: Refusing update from remote peer c5e21c75294b4738be7f6c026cb25341: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:21.433640 28330 raft_consensus.cc:1261] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Refusing update from remote peer c5e21c75294b4738be7f6c026cb25341: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:21.434501 28456 consensus_queue.cc:1035] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:21.435439 28558 consensus_queue.cc:1035] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:21.436353 28469 consensus_queue.cc:1035] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:21.445333 28404 raft_consensus.cc:2932] T 268d32d69f8d4591b72359673a8d9b92 P fb0239de896a4dfeaf520198010246fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 040d1626e04e4dc1b63a334540590bfb (127.24.69.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } attrs { replace: true } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } } }
I20240501 20:01:21.446650 28329 raft_consensus.cc:2932] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 040d1626e04e4dc1b63a334540590bfb (127.24.69.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } attrs { replace: true } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } } }
I20240501 20:01:21.457041 28153 catalog_manager.cc:5484] T 268d32d69f8d4591b72359673a8d9b92 P fb0239de896a4dfeaf520198010246fb reported cstate change: config changed from index 2 to 3, 040d1626e04e4dc1b63a334540590bfb (127.24.69.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c5e21c75294b4738be7f6c026cb25341" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } attrs { replace: true } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } } }
I20240501 20:01:21.670248 28469 raft_consensus.cc:2932] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 040d1626e04e4dc1b63a334540590bfb (127.24.69.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } attrs { replace: true } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } } }
I20240501 20:01:21.670975 28520 raft_consensus.cc:2932] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 040d1626e04e4dc1b63a334540590bfb (127.24.69.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } attrs { replace: true } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } } }
I20240501 20:01:21.735026 28255 consensus_queue.cc:237] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } }
I20240501 20:01:21.739408 28520 raft_consensus.cc:1261] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [term 1 FOLLOWER]: Refusing update from remote peer c5e21c75294b4738be7f6c026cb25341: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:01:21.740096 28329 raft_consensus.cc:1261] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Refusing update from remote peer c5e21c75294b4738be7f6c026cb25341: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:01:21.740685 28456 consensus_queue.cc:1035] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:21.741601 28558 consensus_queue.cc:1035] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:21.747998 28568 raft_consensus.cc:1050] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb: attempting to promote NON_VOTER 040d1626e04e4dc1b63a334540590bfb to VOTER
I20240501 20:01:21.750072 28568 consensus_queue.cc:237] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } attrs { replace: true } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } }
I20240501 20:01:21.749745 28520 raft_consensus.cc:2932] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fb0239de896a4dfeaf520198010246fb (127.24.69.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } } }
I20240501 20:01:21.753952 28330 raft_consensus.cc:2932] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fb0239de896a4dfeaf520198010246fb (127.24.69.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } } }
I20240501 20:01:21.758245 28329 raft_consensus.cc:1261] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Refusing update from remote peer fb0239de896a4dfeaf520198010246fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:21.758287 28255 raft_consensus.cc:1261] T bd534f7d16d44a478ec5f954776b93aa P c5e21c75294b4738be7f6c026cb25341 [term 1 FOLLOWER]: Refusing update from remote peer fb0239de896a4dfeaf520198010246fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:21.759853 28568 consensus_queue.cc:1035] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:21.760775 28455 consensus_queue.cc:1035] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:21.761533 28520 raft_consensus.cc:1261] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb [term 1 LEARNER]: Refusing update from remote peer fb0239de896a4dfeaf520198010246fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:21.762984 28568 consensus_queue.cc:1035] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:21.763990 28153 catalog_manager.cc:5484] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb reported cstate change: config changed from index 3 to 4, VOTER fb0239de896a4dfeaf520198010246fb (127.24.69.131) evicted. New cstate: current_term: 1 leader_uuid: "c5e21c75294b4738be7f6c026cb25341" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } } }
I20240501 20:01:21.775370 28519 raft_consensus.cc:2932] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 040d1626e04e4dc1b63a334540590bfb (127.24.69.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } attrs { replace: true } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } } }
I20240501 20:01:21.777318 28329 raft_consensus.cc:2932] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 040d1626e04e4dc1b63a334540590bfb (127.24.69.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } attrs { replace: true } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } } }
I20240501 20:01:21.876056 28393 tablet_service.cc:1514] Processing DeleteTablet for tablet 268d32d69f8d4591b72359673a8d9b92 with delete_type TABLET_DATA_TOMBSTONED (TS fb0239de896a4dfeaf520198010246fb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58794
I20240501 20:01:21.877164 28455 raft_consensus.cc:2932] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 040d1626e04e4dc1b63a334540590bfb (127.24.69.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } attrs { replace: true } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } } }
I20240501 20:01:21.880033 28592 tablet_replica.cc:330] T 268d32d69f8d4591b72359673a8d9b92 P fb0239de896a4dfeaf520198010246fb: stopping tablet replica
I20240501 20:01:21.880955 28592 raft_consensus.cc:2229] T 268d32d69f8d4591b72359673a8d9b92 P fb0239de896a4dfeaf520198010246fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:21.881850 28592 raft_consensus.cc:2258] T 268d32d69f8d4591b72359673a8d9b92 P fb0239de896a4dfeaf520198010246fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:21.885071 28592 ts_tablet_manager.cc:1875] T 268d32d69f8d4591b72359673a8d9b92 P fb0239de896a4dfeaf520198010246fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:01:21.886030 28152 catalog_manager.cc:5484] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405 reported cstate change: config changed from index 2 to 3, 040d1626e04e4dc1b63a334540590bfb (127.24.69.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb0239de896a4dfeaf520198010246fb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } attrs { replace: true } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } } }
I20240501 20:01:21.891115 28255 raft_consensus.cc:2932] T bd534f7d16d44a478ec5f954776b93aa P c5e21c75294b4738be7f6c026cb25341 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 040d1626e04e4dc1b63a334540590bfb (127.24.69.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } attrs { replace: true } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } } }
I20240501 20:01:21.899749 28592 ts_tablet_manager.cc:1888] T 268d32d69f8d4591b72359673a8d9b92 P fb0239de896a4dfeaf520198010246fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:01:21.900249 28592 log.cc:1196] T 268d32d69f8d4591b72359673a8d9b92 P fb0239de896a4dfeaf520198010246fb: Deleting WAL directory at /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/ts-2-root/wals/268d32d69f8d4591b72359673a8d9b92
I20240501 20:01:21.902077 28139 catalog_manager.cc:4830] TS fb0239de896a4dfeaf520198010246fb (127.24.69.131:35283): tablet 268d32d69f8d4591b72359673a8d9b92 (table test-workload [id=72bed35560fc44488467649b554bc037]) successfully deleted
I20240501 20:01:22.038431 28403 consensus_queue.cc:237] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } }
I20240501 20:01:22.042693 28519 raft_consensus.cc:1261] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb [term 1 FOLLOWER]: Refusing update from remote peer fb0239de896a4dfeaf520198010246fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:01:22.042752 28329 raft_consensus.cc:1261] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Refusing update from remote peer fb0239de896a4dfeaf520198010246fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:01:22.044049 28568 consensus_queue.cc:1035] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:22.044880 28471 consensus_queue.cc:1035] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:22.053468 28329 raft_consensus.cc:2932] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5e21c75294b4738be7f6c026cb25341 (127.24.69.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } } }
I20240501 20:01:22.053726 28519 raft_consensus.cc:2932] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5e21c75294b4738be7f6c026cb25341 (127.24.69.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } } }
I20240501 20:01:22.065059 28150 catalog_manager.cc:5484] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb reported cstate change: config changed from index 3 to 4, VOTER c5e21c75294b4738be7f6c026cb25341 (127.24.69.129) evicted. New cstate: current_term: 1 leader_uuid: "fb0239de896a4dfeaf520198010246fb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } } }
I20240501 20:01:22.186311 28456 raft_consensus.cc:2932] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fb0239de896a4dfeaf520198010246fb (127.24.69.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } } }
I20240501 20:01:22.192871 28140 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 268d32d69f8d4591b72359673a8d9b92 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:01:22.269738 28245 tablet_service.cc:1514] Processing DeleteTablet for tablet bd534f7d16d44a478ec5f954776b93aa with delete_type TABLET_DATA_TOMBSTONED (TS c5e21c75294b4738be7f6c026cb25341 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45766
I20240501 20:01:22.271551 28593 tablet_replica.cc:330] T bd534f7d16d44a478ec5f954776b93aa P c5e21c75294b4738be7f6c026cb25341: stopping tablet replica
I20240501 20:01:22.272537 28593 raft_consensus.cc:2229] T bd534f7d16d44a478ec5f954776b93aa P c5e21c75294b4738be7f6c026cb25341 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:22.273263 28593 raft_consensus.cc:2258] T bd534f7d16d44a478ec5f954776b93aa P c5e21c75294b4738be7f6c026cb25341 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:22.275983 28593 ts_tablet_manager.cc:1875] T bd534f7d16d44a478ec5f954776b93aa P c5e21c75294b4738be7f6c026cb25341: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:01:22.288650 28593 ts_tablet_manager.cc:1888] T bd534f7d16d44a478ec5f954776b93aa P c5e21c75294b4738be7f6c026cb25341: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:01:22.288991 28593 log.cc:1196] T bd534f7d16d44a478ec5f954776b93aa P c5e21c75294b4738be7f6c026cb25341: Deleting WAL directory at /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714593576104309-24854-0/minicluster-data/ts-0-root/wals/bd534f7d16d44a478ec5f954776b93aa
I20240501 20:01:22.290360 28140 catalog_manager.cc:4830] TS c5e21c75294b4738be7f6c026cb25341 (127.24.69.129:37477): tablet bd534f7d16d44a478ec5f954776b93aa (table test-workload [id=72bed35560fc44488467649b554bc037]) successfully deleted
I20240501 20:01:22.448195 28206 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:22.451977 28206 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:01:22.452591 28206 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:01:22.509572 28568 raft_consensus.cc:2932] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5e21c75294b4738be7f6c026cb25341 (127.24.69.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } } }
I20240501 20:01:22.515827 28139 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bd534f7d16d44a478ec5f954776b93aa with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:01:24.453516 28206 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:24.456068 28206 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:01:24.456423 28206 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:01:26.457226 28206 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:26.460326 28206 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:01:26.460660 28206 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:01:28.105738 28596 raft_consensus.cc:482] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5e21c75294b4738be7f6c026cb25341)
I20240501 20:01:28.107637 28596 raft_consensus.cc:504] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } }
W20240501 20:01:28.134583 28276 debug-util.cc:398] Leaking SignalData structure 0x7b08001f4a60 after lost signal to thread 24857
I20240501 20:01:28.135725 28596 leader_election.cc:288] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5e21c75294b4738be7f6c026cb25341 (127.24.69.129:37477), 040d1626e04e4dc1b63a334540590bfb (127.24.69.132:45365)
I20240501 20:01:28.122856 28594 raft_consensus.cc:482] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e6105cadd8345f69404f840588e9405)
I20240501 20:01:28.137243 28597 raft_consensus.cc:482] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fb0239de896a4dfeaf520198010246fb)
I20240501 20:01:28.137418 28594 raft_consensus.cc:504] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } }
I20240501 20:01:28.138103 28597 raft_consensus.cc:504] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } }
I20240501 20:01:28.143059 28598 raft_consensus.cc:482] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e6105cadd8345f69404f840588e9405)
I20240501 20:01:28.144124 28597 leader_election.cc:288] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fb0239de896a4dfeaf520198010246fb (127.24.69.131:35283), 040d1626e04e4dc1b63a334540590bfb (127.24.69.132:45365)
I20240501 20:01:28.145269 28598 raft_consensus.cc:504] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } }
I20240501 20:01:28.148152 28598 leader_election.cc:288] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fb0239de896a4dfeaf520198010246fb (127.24.69.131:35283), 9e6105cadd8345f69404f840588e9405 (127.24.69.130:44491)
I20240501 20:01:28.148370 28403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd534f7d16d44a478ec5f954776b93aa" candidate_uuid: "9e6105cadd8345f69404f840588e9405" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fb0239de896a4dfeaf520198010246fb" is_pre_election: true
I20240501 20:01:28.151037 28594 leader_election.cc:288] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5e21c75294b4738be7f6c026cb25341 (127.24.69.129:37477), 9e6105cadd8345f69404f840588e9405 (127.24.69.130:44491)
I20240501 20:01:28.162256 28403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cb7f9edf2145daabf966b08220b264" candidate_uuid: "c5e21c75294b4738be7f6c026cb25341" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fb0239de896a4dfeaf520198010246fb" is_pre_election: true
I20240501 20:01:28.162421 28255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cb7f9edf2145daabf966b08220b264" candidate_uuid: "fb0239de896a4dfeaf520198010246fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5e21c75294b4738be7f6c026cb25341" is_pre_election: true
I20240501 20:01:28.162937 28255 raft_consensus.cc:2454] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb0239de896a4dfeaf520198010246fb in term 2.
I20240501 20:01:28.167358 28370 leader_election.cc:302] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5e21c75294b4738be7f6c026cb25341, fb0239de896a4dfeaf520198010246fb; no voters:
I20240501 20:01:28.172845 28594 raft_consensus.cc:2781] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:28.173322 28594 raft_consensus.cc:482] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9e6105cadd8345f69404f840588e9405)
I20240501 20:01:28.173746 28594 raft_consensus.cc:3037] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:28.181463 28255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "268d32d69f8d4591b72359673a8d9b92" candidate_uuid: "9e6105cadd8345f69404f840588e9405" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5e21c75294b4738be7f6c026cb25341" is_pre_election: true
I20240501 20:01:28.186002 28601 raft_consensus.cc:482] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fb0239de896a4dfeaf520198010246fb)
I20240501 20:01:28.186916 28601 raft_consensus.cc:504] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } }
I20240501 20:01:28.190654 28601 leader_election.cc:288] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9e6105cadd8345f69404f840588e9405 (127.24.69.130:44491), fb0239de896a4dfeaf520198010246fb (127.24.69.131:35283)
I20240501 20:01:28.199234 28403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd534f7d16d44a478ec5f954776b93aa" candidate_uuid: "040d1626e04e4dc1b63a334540590bfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fb0239de896a4dfeaf520198010246fb" is_pre_election: true
W20240501 20:01:28.202131 28276 debug-util.cc:398] Leaking SignalData structure 0x7b0800035960 after lost signal to thread 28187
W20240501 20:01:28.209861 28276 debug-util.cc:398] Leaking SignalData structure 0x7b08000af980 after lost signal to thread 28279
W20240501 20:01:28.212698 28276 debug-util.cc:398] Leaking SignalData structure 0x7b08000e8160 after lost signal to thread 28428
I20240501 20:01:28.214677 28330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cb7f9edf2145daabf966b08220b264" candidate_uuid: "c5e21c75294b4738be7f6c026cb25341" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9e6105cadd8345f69404f840588e9405" is_pre_election: true
I20240501 20:01:28.215462 28328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cb7f9edf2145daabf966b08220b264" candidate_uuid: "fb0239de896a4dfeaf520198010246fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9e6105cadd8345f69404f840588e9405" is_pre_election: true
I20240501 20:01:28.224908 28219 leader_election.cc:302] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5e21c75294b4738be7f6c026cb25341; no voters: 9e6105cadd8345f69404f840588e9405, fb0239de896a4dfeaf520198010246fb
I20240501 20:01:28.226717 28456 raft_consensus.cc:2726] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:28.234962 28594 raft_consensus.cc:504] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } }
I20240501 20:01:28.243738 28255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cb7f9edf2145daabf966b08220b264" candidate_uuid: "fb0239de896a4dfeaf520198010246fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5e21c75294b4738be7f6c026cb25341"
I20240501 20:01:28.243978 28328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cb7f9edf2145daabf966b08220b264" candidate_uuid: "fb0239de896a4dfeaf520198010246fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9e6105cadd8345f69404f840588e9405"
I20240501 20:01:28.244341 28594 leader_election.cc:288] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [CANDIDATE]: Term 3 election: Requested vote from peers c5e21c75294b4738be7f6c026cb25341 (127.24.69.129:37477), 9e6105cadd8345f69404f840588e9405 (127.24.69.130:44491)
I20240501 20:01:28.245455 28370 leader_election.cc:302] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb0239de896a4dfeaf520198010246fb; no voters: 9e6105cadd8345f69404f840588e9405, c5e21c75294b4738be7f6c026cb25341
I20240501 20:01:28.246317 28594 raft_consensus.cc:2726] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 20:01:28.249912 28328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd534f7d16d44a478ec5f954776b93aa" candidate_uuid: "040d1626e04e4dc1b63a334540590bfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e6105cadd8345f69404f840588e9405" is_pre_election: true
I20240501 20:01:28.250980 28519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "268d32d69f8d4591b72359673a8d9b92" candidate_uuid: "9e6105cadd8345f69404f840588e9405" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "040d1626e04e4dc1b63a334540590bfb" is_pre_election: true
I20240501 20:01:28.251212 28520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd534f7d16d44a478ec5f954776b93aa" candidate_uuid: "9e6105cadd8345f69404f840588e9405" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "040d1626e04e4dc1b63a334540590bfb" is_pre_election: true
I20240501 20:01:28.252756 28296 leader_election.cc:302] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e6105cadd8345f69404f840588e9405; no voters: 040d1626e04e4dc1b63a334540590bfb, c5e21c75294b4738be7f6c026cb25341
I20240501 20:01:28.252861 28484 leader_election.cc:302] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 040d1626e04e4dc1b63a334540590bfb; no voters: 9e6105cadd8345f69404f840588e9405, fb0239de896a4dfeaf520198010246fb
I20240501 20:01:28.253391 28596 raft_consensus.cc:2726] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:28.253849 28296 leader_election.cc:302] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e6105cadd8345f69404f840588e9405; no voters: 040d1626e04e4dc1b63a334540590bfb, fb0239de896a4dfeaf520198010246fb
I20240501 20:01:28.254041 28601 raft_consensus.cc:2726] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:28.254595 28596 raft_consensus.cc:2726] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:28.435400 28403 raft_consensus.cc:1226] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [term 3 FOLLOWER]: Rejecting Update request from peer 9e6105cadd8345f69404f840588e9405 for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:28.436754 28596 consensus_queue.cc:1046] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:28.461745 28206 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:28.465816 28206 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:01:28.466286 28206 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:01:28.531802 28597 raft_consensus.cc:3032] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:28.532222 28597 raft_consensus.cc:729] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9e6105cadd8345f69404f840588e9405, State: Running, Role: LEADER
I20240501 20:01:28.533113 28597 consensus_queue.cc:260] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } }
I20240501 20:01:28.534160 28597 raft_consensus.cc:3037] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:28.764545 24854 tablet_server.cc:178] TabletServer@127.24.69.129:0 shutting down...
I20240501 20:01:28.796641 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:01:28.797528 24854 tablet_replica.cc:330] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341: stopping tablet replica
I20240501 20:01:28.798238 24854 raft_consensus.cc:2229] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:01:28.799162 24854 raft_consensus.cc:2258] T 268d32d69f8d4591b72359673a8d9b92 P c5e21c75294b4738be7f6c026cb25341 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:28.801736 24854 tablet_replica.cc:330] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341: stopping tablet replica
I20240501 20:01:28.802186 24854 raft_consensus.cc:2229] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:28.802644 24854 raft_consensus.cc:2258] T 20cb7f9edf2145daabf966b08220b264 P c5e21c75294b4738be7f6c026cb25341 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:29.293576 28607 raft_consensus.cc:482] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:29.294004 28607 raft_consensus.cc:504] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } }
I20240501 20:01:29.295529 28607 leader_election.cc:288] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5e21c75294b4738be7f6c026cb25341 (127.24.69.129:37477), fb0239de896a4dfeaf520198010246fb (127.24.69.131:35283)
I20240501 20:01:29.296726 28403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cb7f9edf2145daabf966b08220b264" candidate_uuid: "9e6105cadd8345f69404f840588e9405" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fb0239de896a4dfeaf520198010246fb" is_pre_election: true
I20240501 20:01:29.297394 28403 raft_consensus.cc:2454] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e6105cadd8345f69404f840588e9405 in term 3.
W20240501 20:01:29.297799 28296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.69.129:37477: connect: Connection refused (error 111) [suppressed 3 similar messages]
I20240501 20:01:29.298461 28295 leader_election.cc:302] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e6105cadd8345f69404f840588e9405, fb0239de896a4dfeaf520198010246fb; no voters:
I20240501 20:01:29.299196 28607 raft_consensus.cc:2781] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:29.299588 28607 raft_consensus.cc:482] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:29.299911 28607 raft_consensus.cc:3037] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [term 3 FOLLOWER]: Advancing to term 4
W20240501 20:01:29.301599 28296 leader_election.cc:334] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c5e21c75294b4738be7f6c026cb25341 (127.24.69.129:37477): Network error: Client connection negotiation failed: client connection to 127.24.69.129:37477: connect: Connection refused (error 111)
I20240501 20:01:29.305470 28607 raft_consensus.cc:504] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } }
I20240501 20:01:29.306944 28607 leader_election.cc:288] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [CANDIDATE]: Term 4 election: Requested vote from peers c5e21c75294b4738be7f6c026cb25341 (127.24.69.129:37477), fb0239de896a4dfeaf520198010246fb (127.24.69.131:35283)
I20240501 20:01:29.307888 28403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cb7f9edf2145daabf966b08220b264" candidate_uuid: "9e6105cadd8345f69404f840588e9405" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fb0239de896a4dfeaf520198010246fb"
I20240501 20:01:29.308365 28403 raft_consensus.cc:3037] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [term 3 FOLLOWER]: Advancing to term 4
W20240501 20:01:29.310945 28296 leader_election.cc:334] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer c5e21c75294b4738be7f6c026cb25341 (127.24.69.129:37477): Network error: Client connection negotiation failed: client connection to 127.24.69.129:37477: connect: Connection refused (error 111)
I20240501 20:01:29.313655 28403 raft_consensus.cc:2454] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e6105cadd8345f69404f840588e9405 in term 4.
I20240501 20:01:29.314476 28295 leader_election.cc:302] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9e6105cadd8345f69404f840588e9405, fb0239de896a4dfeaf520198010246fb; no voters: c5e21c75294b4738be7f6c026cb25341
I20240501 20:01:29.315088 28607 raft_consensus.cc:2781] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:29.316072 28607 raft_consensus.cc:686] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [term 4 LEADER]: Becoming Leader. State: Replica: 9e6105cadd8345f69404f840588e9405, State: Running, Role: LEADER
I20240501 20:01:29.316924 28607 consensus_queue.cc:237] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } }
I20240501 20:01:29.327754 28151 catalog_manager.cc:5484] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "9e6105cadd8345f69404f840588e9405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:29.496448 28607 raft_consensus.cc:482] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5e21c75294b4738be7f6c026cb25341)
I20240501 20:01:29.496934 28607 raft_consensus.cc:504] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } }
I20240501 20:01:29.498469 28607 leader_election.cc:288] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5e21c75294b4738be7f6c026cb25341 (127.24.69.129:37477), 040d1626e04e4dc1b63a334540590bfb (127.24.69.132:45365)
I20240501 20:01:29.500286 28519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "268d32d69f8d4591b72359673a8d9b92" candidate_uuid: "9e6105cadd8345f69404f840588e9405" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "040d1626e04e4dc1b63a334540590bfb" is_pre_election: true
W20240501 20:01:29.502044 28296 leader_election.cc:334] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c5e21c75294b4738be7f6c026cb25341 (127.24.69.129:37477): Network error: Client connection negotiation failed: client connection to 127.24.69.129:37477: connect: Connection refused (error 111)
I20240501 20:01:29.502410 28296 leader_election.cc:302] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e6105cadd8345f69404f840588e9405; no voters: 040d1626e04e4dc1b63a334540590bfb, c5e21c75294b4738be7f6c026cb25341
I20240501 20:01:29.502982 28607 raft_consensus.cc:2726] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:29.560904 28403 raft_consensus.cc:1261] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [term 4 FOLLOWER]: Refusing update from remote peer 9e6105cadd8345f69404f840588e9405: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:29.562105 28607 consensus_queue.cc:1035] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0239de896a4dfeaf520198010246fb" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 35283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:29.643514 28618 raft_consensus.cc:482] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5e21c75294b4738be7f6c026cb25341)
I20240501 20:01:29.644033 28618 raft_consensus.cc:504] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } }
I20240501 20:01:29.646075 28618 leader_election.cc:288] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5e21c75294b4738be7f6c026cb25341 (127.24.69.129:37477), 9e6105cadd8345f69404f840588e9405 (127.24.69.130:44491)
I20240501 20:01:29.647372 28328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "268d32d69f8d4591b72359673a8d9b92" candidate_uuid: "040d1626e04e4dc1b63a334540590bfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e6105cadd8345f69404f840588e9405" is_pre_election: true
I20240501 20:01:29.647986 28328 raft_consensus.cc:2454] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 040d1626e04e4dc1b63a334540590bfb in term 1.
I20240501 20:01:29.649091 28484 leader_election.cc:302] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 040d1626e04e4dc1b63a334540590bfb, 9e6105cadd8345f69404f840588e9405; no voters:
I20240501 20:01:29.650218 28618 raft_consensus.cc:2781] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:29.651844 28618 raft_consensus.cc:482] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader c5e21c75294b4738be7f6c026cb25341)
W20240501 20:01:29.652503 28486 leader_election.cc:334] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c5e21c75294b4738be7f6c026cb25341 (127.24.69.129:37477): Network error: Client connection negotiation failed: client connection to 127.24.69.129:37477: connect: Connection refused (error 111)
I20240501 20:01:29.652242 28618 raft_consensus.cc:3037] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:29.658581 28618 raft_consensus.cc:504] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } }
I20240501 20:01:29.660138 28618 leader_election.cc:288] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [CANDIDATE]: Term 2 election: Requested vote from peers c5e21c75294b4738be7f6c026cb25341 (127.24.69.129:37477), 9e6105cadd8345f69404f840588e9405 (127.24.69.130:44491)
I20240501 20:01:29.661288 28328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "268d32d69f8d4591b72359673a8d9b92" candidate_uuid: "040d1626e04e4dc1b63a334540590bfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9e6105cadd8345f69404f840588e9405"
I20240501 20:01:29.661870 28328 raft_consensus.cc:3037] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:01:29.663940 28486 leader_election.cc:334] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c5e21c75294b4738be7f6c026cb25341 (127.24.69.129:37477): Network error: Client connection negotiation failed: client connection to 127.24.69.129:37477: connect: Connection refused (error 111)
I20240501 20:01:29.667208 28328 raft_consensus.cc:2454] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 040d1626e04e4dc1b63a334540590bfb in term 2.
I20240501 20:01:29.668046 28484 leader_election.cc:302] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 040d1626e04e4dc1b63a334540590bfb, 9e6105cadd8345f69404f840588e9405; no voters: c5e21c75294b4738be7f6c026cb25341
I20240501 20:01:29.668727 28618 raft_consensus.cc:2781] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:29.670017 28618 raft_consensus.cc:686] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [term 2 LEADER]: Becoming Leader. State: Replica: 040d1626e04e4dc1b63a334540590bfb, State: Running, Role: LEADER
I20240501 20:01:29.671262 28618 consensus_queue.cc:237] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } }
I20240501 20:01:29.679386 28153 catalog_manager.cc:5484] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb reported cstate change: term changed from 1 to 2, leader changed from c5e21c75294b4738be7f6c026cb25341 (127.24.69.129) to 040d1626e04e4dc1b63a334540590bfb (127.24.69.132). New cstate: current_term: 2 leader_uuid: "040d1626e04e4dc1b63a334540590bfb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5e21c75294b4738be7f6c026cb25341" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 37477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "040d1626e04e4dc1b63a334540590bfb" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 45365 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 20:01:29.685904 28296 consensus_peers.cc:485] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 -> Peer c5e21c75294b4738be7f6c026cb25341 (127.24.69.129:37477): Couldn't send request to peer c5e21c75294b4738be7f6c026cb25341. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:37477: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:01:29.816134 24854 thread.cc:536] Waited for 1000ms trying to join with diag-logger (tid 28276)
W20240501 20:01:29.909948 28486 consensus_peers.cc:485] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb -> Peer c5e21c75294b4738be7f6c026cb25341 (127.24.69.129:37477): Couldn't send request to peer c5e21c75294b4738be7f6c026cb25341. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:37477: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:29.986527 28328 raft_consensus.cc:1261] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [term 2 FOLLOWER]: Refusing update from remote peer 040d1626e04e4dc1b63a334540590bfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:01:29.987807 28618 consensus_queue.cc:1035] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e6105cadd8345f69404f840588e9405" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 44491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:01:30.467214 28206 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:30.469935 28206 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:01:30.470305 28206 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240501 20:01:30.816651 24854 thread.cc:536] Waited for 2000ms trying to join with diag-logger (tid 28276)
I20240501 20:01:31.035615 24854 tablet_server.cc:194] TabletServer@127.24.69.129:0 shutdown complete.
I20240501 20:01:31.050575 24854 tablet_server.cc:178] TabletServer@127.24.69.130:0 shutting down...
I20240501 20:01:31.071767 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:01:31.072508 24854 tablet_replica.cc:330] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405: stopping tablet replica
I20240501 20:01:31.073405 24854 raft_consensus.cc:2229] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240501 20:01:31.073980 28484 consensus_peers.cc:485] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb -> Peer 9e6105cadd8345f69404f840588e9405 (127.24.69.130:44491): Couldn't send request to peer 9e6105cadd8345f69404f840588e9405. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:31.074075 24854 raft_consensus.cc:2258] T bd534f7d16d44a478ec5f954776b93aa P 9e6105cadd8345f69404f840588e9405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:31.076184 24854 tablet_replica.cc:330] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405: stopping tablet replica
I20240501 20:01:31.076737 24854 raft_consensus.cc:2229] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [term 4 LEADER]: Raft consensus shutting down.
I20240501 20:01:31.077544 24854 raft_consensus.cc:2258] T 20cb7f9edf2145daabf966b08220b264 P 9e6105cadd8345f69404f840588e9405 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:31.079373 24854 tablet_replica.cc:330] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405: stopping tablet replica
I20240501 20:01:31.079813 24854 raft_consensus.cc:2229] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:31.080492 24854 raft_consensus.cc:2258] T 268d32d69f8d4591b72359673a8d9b92 P 9e6105cadd8345f69404f840588e9405 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:31.101948 24854 tablet_server.cc:194] TabletServer@127.24.69.130:0 shutdown complete.
I20240501 20:01:31.117048 24854 tablet_server.cc:178] TabletServer@127.24.69.131:0 shutting down...
W20240501 20:01:31.141206 28368 consensus_peers.cc:485] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb -> Peer 9e6105cadd8345f69404f840588e9405 (127.24.69.130:44491): Couldn't send request to peer 9e6105cadd8345f69404f840588e9405. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.130:44491: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:31.143414 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:01:31.144109 24854 tablet_replica.cc:330] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb: stopping tablet replica
I20240501 20:01:31.144764 24854 raft_consensus.cc:2229] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:31.145188 24854 raft_consensus.cc:2258] T 20cb7f9edf2145daabf966b08220b264 P fb0239de896a4dfeaf520198010246fb [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:31.147207 24854 tablet_replica.cc:330] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb: stopping tablet replica
I20240501 20:01:31.147636 24854 raft_consensus.cc:2229] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:01:31.148454 24854 raft_consensus.cc:2258] T bd534f7d16d44a478ec5f954776b93aa P fb0239de896a4dfeaf520198010246fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:31.171162 24854 tablet_server.cc:194] TabletServer@127.24.69.131:0 shutdown complete.
I20240501 20:01:31.184468 24854 tablet_server.cc:178] TabletServer@127.24.69.132:0 shutting down...
I20240501 20:01:31.205896 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:01:31.206477 24854 tablet_replica.cc:330] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb: stopping tablet replica
I20240501 20:01:31.206998 24854 raft_consensus.cc:2229] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:31.207526 24854 raft_consensus.cc:2258] T bd534f7d16d44a478ec5f954776b93aa P 040d1626e04e4dc1b63a334540590bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:31.209424 24854 tablet_replica.cc:330] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb: stopping tablet replica
I20240501 20:01:31.209897 24854 raft_consensus.cc:2229] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [term 2 LEADER]: Raft consensus shutting down.
I20240501 20:01:31.210774 24854 raft_consensus.cc:2258] T 268d32d69f8d4591b72359673a8d9b92 P 040d1626e04e4dc1b63a334540590bfb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:31.230113 24854 tablet_server.cc:194] TabletServer@127.24.69.132:0 shutdown complete.
I20240501 20:01:31.240681 24854 master.cc:537] Master@127.24.69.190:34921 shutting down...
I20240501 20:01:31.257884 24854 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fb27659c76a544b0a89c0f05bf499012 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:01:31.258364 24854 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fb27659c76a544b0a89c0f05bf499012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:31.258637 24854 tablet_replica.cc:330] T 00000000000000000000000000000000 P fb27659c76a544b0a89c0f05bf499012: stopping tablet replica
I20240501 20:01:31.276093 24854 master.cc:552] Master@127.24.69.190:34921 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingUnstableCluster (13044 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor
I20240501 20:01:31.312059 24854 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.69.190:36253
I20240501 20:01:31.313136 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:31.319368 28631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:31.319949 28632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:31.320608 28634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:31.321323 24854 server_base.cc:1034] running on GCE node
I20240501 20:01:31.322037 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:01:31.322232 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:31.322366 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593691322354 us; error 0 us; skew 500 ppm
I20240501 20:01:31.322779 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:31.329008 24854 webserver.cc:460] Webserver started at http://127.24.69.190:39227/ using document root <none> and password file <none>
I20240501 20:01:31.329427 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:31.329582 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:31.329804 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:31.330914 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714593576104309-24854-0/minicluster-data/master-0-root/instance:
uuid: "07f966c955ab4f899f50206b048f68b2"
format_stamp: "Formatted at 2024-05-01 20:01:31 on dist-test-slave-z8x3"
I20240501 20:01:31.335191 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 20:01:31.338351 28639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:31.339059 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:01:31.339334 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714593576104309-24854-0/minicluster-data/master-0-root
uuid: "07f966c955ab4f899f50206b048f68b2"
format_stamp: "Formatted at 2024-05-01 20:01:31 on dist-test-slave-z8x3"
I20240501 20:01:31.339596 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714593576104309-24854-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714593576104309-24854-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714593576104309-24854-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:31.365516 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:31.366640 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:31.402588 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.190:36253
I20240501 20:01:31.402679 28690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.190:36253 every 8 connection(s)
I20240501 20:01:31.406301 28691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:31.416078 28691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07f966c955ab4f899f50206b048f68b2: Bootstrap starting.
I20240501 20:01:31.420115 28691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07f966c955ab4f899f50206b048f68b2: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:31.423902 28691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07f966c955ab4f899f50206b048f68b2: No bootstrap required, opened a new log
I20240501 20:01:31.425769 28691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07f966c955ab4f899f50206b048f68b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f966c955ab4f899f50206b048f68b2" member_type: VOTER }
I20240501 20:01:31.426133 28691 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07f966c955ab4f899f50206b048f68b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:31.426309 28691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07f966c955ab4f899f50206b048f68b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07f966c955ab4f899f50206b048f68b2, State: Initialized, Role: FOLLOWER
I20240501 20:01:31.426749 28691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07f966c955ab4f899f50206b048f68b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f966c955ab4f899f50206b048f68b2" member_type: VOTER }
I20240501 20:01:31.427168 28691 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07f966c955ab4f899f50206b048f68b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:31.427352 28691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07f966c955ab4f899f50206b048f68b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:31.427551 28691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07f966c955ab4f899f50206b048f68b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:31.431550 28691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07f966c955ab4f899f50206b048f68b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f966c955ab4f899f50206b048f68b2" member_type: VOTER }
I20240501 20:01:31.431999 28691 leader_election.cc:302] T 00000000000000000000000000000000 P 07f966c955ab4f899f50206b048f68b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07f966c955ab4f899f50206b048f68b2; no voters:
I20240501 20:01:31.433039 28691 leader_election.cc:288] T 00000000000000000000000000000000 P 07f966c955ab4f899f50206b048f68b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:31.433418 28694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07f966c955ab4f899f50206b048f68b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:31.434686 28694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07f966c955ab4f899f50206b048f68b2 [term 1 LEADER]: Becoming Leader. State: Replica: 07f966c955ab4f899f50206b048f68b2, State: Running, Role: LEADER
I20240501 20:01:31.435359 28694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07f966c955ab4f899f50206b048f68b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f966c955ab4f899f50206b048f68b2" member_type: VOTER }
I20240501 20:01:31.435900 28691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07f966c955ab4f899f50206b048f68b2 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:31.439174 28695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07f966c955ab4f899f50206b048f68b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07f966c955ab4f899f50206b048f68b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f966c955ab4f899f50206b048f68b2" member_type: VOTER } }
I20240501 20:01:31.439944 28695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07f966c955ab4f899f50206b048f68b2 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:31.439631 28696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07f966c955ab4f899f50206b048f68b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07f966c955ab4f899f50206b048f68b2. Latest consensus state: current_term: 1 leader_uuid: "07f966c955ab4f899f50206b048f68b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07f966c955ab4f899f50206b048f68b2" member_type: VOTER } }
I20240501 20:01:31.440685 28696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07f966c955ab4f899f50206b048f68b2 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:31.443702 28700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:31.452006 28700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:31.457194 24854 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 20:01:31.461799 28700 catalog_manager.cc:1347] Generated new cluster ID: 9c578fb6209e491c87c857942f35bd5b
I20240501 20:01:31.462075 28700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:31.483377 28700 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:31.484696 28700 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:31.499298 28700 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 07f966c955ab4f899f50206b048f68b2: Generated new TSK 0
I20240501 20:01:31.499883 28700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:01:31.523288 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:31.530092 28712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:31.530521 28713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:31.531738 28715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:31.532523 24854 server_base.cc:1034] running on GCE node
I20240501 20:01:31.533337 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:01:31.533522 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:31.533658 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593691533642 us; error 0 us; skew 500 ppm
I20240501 20:01:31.534139 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:31.536615 24854 webserver.cc:460] Webserver started at http://127.24.69.129:42833/ using document root <none> and password file <none>
I20240501 20:01:31.537096 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:31.537278 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:31.537510 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:31.538564 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714593576104309-24854-0/minicluster-data/ts-0-root/instance:
uuid: "7249ca819fd14f9e92070d573add724a"
format_stamp: "Formatted at 2024-05-01 20:01:31 on dist-test-slave-z8x3"
I20240501 20:01:31.542872 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 20:01:31.545982 28720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:31.546636 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:01:31.546902 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714593576104309-24854-0/minicluster-data/ts-0-root
uuid: "7249ca819fd14f9e92070d573add724a"
format_stamp: "Formatted at 2024-05-01 20:01:31 on dist-test-slave-z8x3"
I20240501 20:01:31.547164 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714593576104309-24854-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714593576104309-24854-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714593576104309-24854-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:31.563511 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:31.564550 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:31.565876 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:31.568007 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:31.568194 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:31.568477 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:31.568640 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:31.606998 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.129:42719
I20240501 20:01:31.607113 28782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.129:42719 every 8 connection(s)
I20240501 20:01:31.611522 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:31.619534 28787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:31.620483 28788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:31.623878 28790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:31.624426 28783 heartbeater.cc:347] Connected to a master server at 127.24.69.190:36253
I20240501 20:01:31.624706 24854 server_base.cc:1034] running on GCE node
I20240501 20:01:31.624796 28783 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:31.625591 28783 heartbeater.cc:511] Master 127.24.69.190:36253 requested a full tablet report, sending...
I20240501 20:01:31.625795 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:01:31.626039 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:31.626183 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593691626165 us; error 0 us; skew 500 ppm
I20240501 20:01:31.626833 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:31.627547 28656 ts_manager.cc:194] Registered new tserver with Master: 7249ca819fd14f9e92070d573add724a (127.24.69.129:42719)
I20240501 20:01:31.629380 24854 webserver.cc:460] Webserver started at http://127.24.69.130:33763/ using document root <none> and password file <none>
I20240501 20:01:31.629607 28656 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47786
I20240501 20:01:31.629843 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:31.630085 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:31.630326 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:31.631438 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714593576104309-24854-0/minicluster-data/ts-1-root/instance:
uuid: "8f2fd7440d99455aa0134d3e7e25c8cb"
format_stamp: "Formatted at 2024-05-01 20:01:31 on dist-test-slave-z8x3"
I20240501 20:01:31.635592 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 20:01:31.638828 28795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:31.639530 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:01:31.639863 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714593576104309-24854-0/minicluster-data/ts-1-root
uuid: "8f2fd7440d99455aa0134d3e7e25c8cb"
format_stamp: "Formatted at 2024-05-01 20:01:31 on dist-test-slave-z8x3"
I20240501 20:01:31.640106 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714593576104309-24854-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714593576104309-24854-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714593576104309-24854-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:31.662441 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:31.663524 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:31.664968 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:31.667076 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:31.667271 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:31.667487 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:31.667636 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:31.703814 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.130:32851
I20240501 20:01:31.703913 28857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.130:32851 every 8 connection(s)
I20240501 20:01:31.708321 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:31.716094 28862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:31.716976 28861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:31.717619 28864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:31.719228 24854 server_base.cc:1034] running on GCE node
I20240501 20:01:31.719900 28858 heartbeater.cc:347] Connected to a master server at 127.24.69.190:36253
I20240501 20:01:31.720098 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240501 20:01:31.720229 28858 heartbeater.cc:464] Registering TS with master...
W20240501 20:01:31.720371 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:31.720594 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593691720576 us; error 0 us; skew 500 ppm
I20240501 20:01:31.721021 28858 heartbeater.cc:511] Master 127.24.69.190:36253 requested a full tablet report, sending...
I20240501 20:01:31.721124 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:31.723062 28656 ts_manager.cc:194] Registered new tserver with Master: 8f2fd7440d99455aa0134d3e7e25c8cb (127.24.69.130:32851)
I20240501 20:01:31.723955 24854 webserver.cc:460] Webserver started at http://127.24.69.131:45881/ using document root <none> and password file <none>
I20240501 20:01:31.724588 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:31.724819 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:31.724963 28656 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47800
I20240501 20:01:31.725135 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:31.726553 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714593576104309-24854-0/minicluster-data/ts-2-root/instance:
uuid: "b81faa4c3db94ac6a8bc782a5a95d2e2"
format_stamp: "Formatted at 2024-05-01 20:01:31 on dist-test-slave-z8x3"
I20240501 20:01:31.730895 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 20:01:31.733901 28869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:31.734582 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:01:31.734884 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714593576104309-24854-0/minicluster-data/ts-2-root
uuid: "b81faa4c3db94ac6a8bc782a5a95d2e2"
format_stamp: "Formatted at 2024-05-01 20:01:31 on dist-test-slave-z8x3"
I20240501 20:01:31.735193 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714593576104309-24854-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714593576104309-24854-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1714593576104309-24854-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:31.749610 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:31.750645 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:31.752008 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:31.754132 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:31.754328 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:31.754530 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:31.754686 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:31.791239 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.131:45213
I20240501 20:01:31.791335 28931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.131:45213 every 8 connection(s)
I20240501 20:01:31.803690 28932 heartbeater.cc:347] Connected to a master server at 127.24.69.190:36253
I20240501 20:01:31.804056 28932 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:31.804802 28932 heartbeater.cc:511] Master 127.24.69.190:36253 requested a full tablet report, sending...
I20240501 20:01:31.806476 28656 ts_manager.cc:194] Registered new tserver with Master: b81faa4c3db94ac6a8bc782a5a95d2e2 (127.24.69.131:45213)
I20240501 20:01:31.807102 24854 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012986158s
I20240501 20:01:31.807770 28656 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47816
I20240501 20:01:32.631937 28783 heartbeater.cc:503] Master 127.24.69.190:36253 was elected leader, sending a full tablet report...
I20240501 20:01:32.727313 28858 heartbeater.cc:503] Master 127.24.69.190:36253 was elected leader, sending a full tablet report...
I20240501 20:01:32.810307 28932 heartbeater.cc:503] Master 127.24.69.190:36253 was elected leader, sending a full tablet report...
I20240501 20:01:32.838855 24854 test_util.cc:268] Using random seed: -201397328
I20240501 20:01:32.860428 28656 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47820:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 20:01:32.862401 28656 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:01:32.910569 28748 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca4d2159b1548a880e9ea07077a8969 (DEFAULT_TABLE table=test-workload [id=7bcda07476f24693a11dc5b0fb64ccad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:01:32.911904 28748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca4d2159b1548a880e9ea07077a8969. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:32.911666 28897 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca4d2159b1548a880e9ea07077a8969 (DEFAULT_TABLE table=test-workload [id=7bcda07476f24693a11dc5b0fb64ccad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:01:32.912878 28897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca4d2159b1548a880e9ea07077a8969. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:32.913533 28823 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca4d2159b1548a880e9ea07077a8969 (DEFAULT_TABLE table=test-workload [id=7bcda07476f24693a11dc5b0fb64ccad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:01:32.914484 28823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca4d2159b1548a880e9ea07077a8969. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:32.927501 28952 tablet_bootstrap.cc:492] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb: Bootstrap starting.
I20240501 20:01:32.928985 28953 tablet_bootstrap.cc:492] T 0ca4d2159b1548a880e9ea07077a8969 P 7249ca819fd14f9e92070d573add724a: Bootstrap starting.
I20240501 20:01:32.930861 28954 tablet_bootstrap.cc:492] T 0ca4d2159b1548a880e9ea07077a8969 P b81faa4c3db94ac6a8bc782a5a95d2e2: Bootstrap starting.
I20240501 20:01:32.933934 28952 tablet_bootstrap.cc:654] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:32.935088 28953 tablet_bootstrap.cc:654] T 0ca4d2159b1548a880e9ea07077a8969 P 7249ca819fd14f9e92070d573add724a: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:32.935825 28954 tablet_bootstrap.cc:654] T 0ca4d2159b1548a880e9ea07077a8969 P b81faa4c3db94ac6a8bc782a5a95d2e2: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:32.939853 28952 tablet_bootstrap.cc:492] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb: No bootstrap required, opened a new log
I20240501 20:01:32.940294 28952 ts_tablet_manager.cc:1367] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240501 20:01:32.941431 28953 tablet_bootstrap.cc:492] T 0ca4d2159b1548a880e9ea07077a8969 P 7249ca819fd14f9e92070d573add724a: No bootstrap required, opened a new log
I20240501 20:01:32.941397 28954 tablet_bootstrap.cc:492] T 0ca4d2159b1548a880e9ea07077a8969 P b81faa4c3db94ac6a8bc782a5a95d2e2: No bootstrap required, opened a new log
I20240501 20:01:32.941968 28953 ts_tablet_manager.cc:1367] T 0ca4d2159b1548a880e9ea07077a8969 P 7249ca819fd14f9e92070d573add724a: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20240501 20:01:32.942049 28954 ts_tablet_manager.cc:1367] T 0ca4d2159b1548a880e9ea07077a8969 P b81faa4c3db94ac6a8bc782a5a95d2e2: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20240501 20:01:32.942440 28952 raft_consensus.cc:348] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2fd7440d99455aa0134d3e7e25c8cb" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 32851 } } peers { permanent_uuid: "7249ca819fd14f9e92070d573add724a" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 42719 } } peers { permanent_uuid: "b81faa4c3db94ac6a8bc782a5a95d2e2" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 45213 } }
I20240501 20:01:32.943315 28952 raft_consensus.cc:374] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:32.943666 28952 raft_consensus.cc:729] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2fd7440d99455aa0134d3e7e25c8cb, State: Initialized, Role: FOLLOWER
I20240501 20:01:32.944432 28952 consensus_queue.cc:260] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2fd7440d99455aa0134d3e7e25c8cb" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 32851 } } peers { permanent_uuid: "7249ca819fd14f9e92070d573add724a" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 42719 } } peers { permanent_uuid: "b81faa4c3db94ac6a8bc782a5a95d2e2" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 45213 } }
I20240501 20:01:32.944573 28954 raft_consensus.cc:348] T 0ca4d2159b1548a880e9ea07077a8969 P b81faa4c3db94ac6a8bc782a5a95d2e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2fd7440d99455aa0134d3e7e25c8cb" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 32851 } } peers { permanent_uuid: "7249ca819fd14f9e92070d573add724a" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 42719 } } peers { permanent_uuid: "b81faa4c3db94ac6a8bc782a5a95d2e2" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 45213 } }
I20240501 20:01:32.944805 28953 raft_consensus.cc:348] T 0ca4d2159b1548a880e9ea07077a8969 P 7249ca819fd14f9e92070d573add724a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2fd7440d99455aa0134d3e7e25c8cb" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 32851 } } peers { permanent_uuid: "7249ca819fd14f9e92070d573add724a" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 42719 } } peers { permanent_uuid: "b81faa4c3db94ac6a8bc782a5a95d2e2" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 45213 } }
I20240501 20:01:32.945358 28954 raft_consensus.cc:374] T 0ca4d2159b1548a880e9ea07077a8969 P b81faa4c3db94ac6a8bc782a5a95d2e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:32.945489 28953 raft_consensus.cc:374] T 0ca4d2159b1548a880e9ea07077a8969 P 7249ca819fd14f9e92070d573add724a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:32.945700 28954 raft_consensus.cc:729] T 0ca4d2159b1548a880e9ea07077a8969 P b81faa4c3db94ac6a8bc782a5a95d2e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81faa4c3db94ac6a8bc782a5a95d2e2, State: Initialized, Role: FOLLOWER
I20240501 20:01:32.945891 28953 raft_consensus.cc:729] T 0ca4d2159b1548a880e9ea07077a8969 P 7249ca819fd14f9e92070d573add724a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7249ca819fd14f9e92070d573add724a, State: Initialized, Role: FOLLOWER
I20240501 20:01:32.946703 28954 consensus_queue.cc:260] T 0ca4d2159b1548a880e9ea07077a8969 P b81faa4c3db94ac6a8bc782a5a95d2e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2fd7440d99455aa0134d3e7e25c8cb" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 32851 } } peers { permanent_uuid: "7249ca819fd14f9e92070d573add724a" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 42719 } } peers { permanent_uuid: "b81faa4c3db94ac6a8bc782a5a95d2e2" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 45213 } }
I20240501 20:01:32.946662 28953 consensus_queue.cc:260] T 0ca4d2159b1548a880e9ea07077a8969 P 7249ca819fd14f9e92070d573add724a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2fd7440d99455aa0134d3e7e25c8cb" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 32851 } } peers { permanent_uuid: "7249ca819fd14f9e92070d573add724a" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 42719 } } peers { permanent_uuid: "b81faa4c3db94ac6a8bc782a5a95d2e2" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 45213 } }
I20240501 20:01:32.948463 28952 ts_tablet_manager.cc:1398] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240501 20:01:32.950873 28954 ts_tablet_manager.cc:1398] T 0ca4d2159b1548a880e9ea07077a8969 P b81faa4c3db94ac6a8bc782a5a95d2e2: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20240501 20:01:32.951556 28953 ts_tablet_manager.cc:1398] T 0ca4d2159b1548a880e9ea07077a8969 P 7249ca819fd14f9e92070d573add724a: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
W20240501 20:01:32.958392 28859 tablet.cc:2369] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:01:33.064393 28958 raft_consensus.cc:482] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:33.064899 28958 raft_consensus.cc:504] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2fd7440d99455aa0134d3e7e25c8cb" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 32851 } } peers { permanent_uuid: "7249ca819fd14f9e92070d573add724a" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 42719 } } peers { permanent_uuid: "b81faa4c3db94ac6a8bc782a5a95d2e2" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 45213 } }
I20240501 20:01:33.066859 28958 leader_election.cc:288] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7249ca819fd14f9e92070d573add724a (127.24.69.129:42719), b81faa4c3db94ac6a8bc782a5a95d2e2 (127.24.69.131:45213)
I20240501 20:01:33.078013 28758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca4d2159b1548a880e9ea07077a8969" candidate_uuid: "8f2fd7440d99455aa0134d3e7e25c8cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7249ca819fd14f9e92070d573add724a" is_pre_election: true
I20240501 20:01:33.078219 28907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca4d2159b1548a880e9ea07077a8969" candidate_uuid: "8f2fd7440d99455aa0134d3e7e25c8cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81faa4c3db94ac6a8bc782a5a95d2e2" is_pre_election: true
I20240501 20:01:33.078831 28758 raft_consensus.cc:2454] T 0ca4d2159b1548a880e9ea07077a8969 P 7249ca819fd14f9e92070d573add724a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f2fd7440d99455aa0134d3e7e25c8cb in term 0.
I20240501 20:01:33.078907 28907 raft_consensus.cc:2454] T 0ca4d2159b1548a880e9ea07077a8969 P b81faa4c3db94ac6a8bc782a5a95d2e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f2fd7440d99455aa0134d3e7e25c8cb in term 0.
I20240501 20:01:33.079818 28797 leader_election.cc:302] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7249ca819fd14f9e92070d573add724a, 8f2fd7440d99455aa0134d3e7e25c8cb; no voters:
I20240501 20:01:33.080477 28958 raft_consensus.cc:2781] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:33.080797 28958 raft_consensus.cc:482] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:33.081012 28958 raft_consensus.cc:3037] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:33.086010 28958 raft_consensus.cc:504] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2fd7440d99455aa0134d3e7e25c8cb" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 32851 } } peers { permanent_uuid: "7249ca819fd14f9e92070d573add724a" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 42719 } } peers { permanent_uuid: "b81faa4c3db94ac6a8bc782a5a95d2e2" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 45213 } }
I20240501 20:01:33.087544 28958 leader_election.cc:288] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb [CANDIDATE]: Term 1 election: Requested vote from peers 7249ca819fd14f9e92070d573add724a (127.24.69.129:42719), b81faa4c3db94ac6a8bc782a5a95d2e2 (127.24.69.131:45213)
I20240501 20:01:33.088332 28758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca4d2159b1548a880e9ea07077a8969" candidate_uuid: "8f2fd7440d99455aa0134d3e7e25c8cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7249ca819fd14f9e92070d573add724a"
I20240501 20:01:33.088557 28907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca4d2159b1548a880e9ea07077a8969" candidate_uuid: "8f2fd7440d99455aa0134d3e7e25c8cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81faa4c3db94ac6a8bc782a5a95d2e2"
I20240501 20:01:33.088804 28758 raft_consensus.cc:3037] T 0ca4d2159b1548a880e9ea07077a8969 P 7249ca819fd14f9e92070d573add724a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:33.089092 28907 raft_consensus.cc:3037] T 0ca4d2159b1548a880e9ea07077a8969 P b81faa4c3db94ac6a8bc782a5a95d2e2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:33.093336 28758 raft_consensus.cc:2454] T 0ca4d2159b1548a880e9ea07077a8969 P 7249ca819fd14f9e92070d573add724a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f2fd7440d99455aa0134d3e7e25c8cb in term 1.
I20240501 20:01:33.093581 28907 raft_consensus.cc:2454] T 0ca4d2159b1548a880e9ea07077a8969 P b81faa4c3db94ac6a8bc782a5a95d2e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f2fd7440d99455aa0134d3e7e25c8cb in term 1.
I20240501 20:01:33.094290 28797 leader_election.cc:302] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7249ca819fd14f9e92070d573add724a, 8f2fd7440d99455aa0134d3e7e25c8cb; no voters:
I20240501 20:01:33.094965 28958 raft_consensus.cc:2781] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:33.095968 28958 raft_consensus.cc:686] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb [term 1 LEADER]: Becoming Leader. State: Replica: 8f2fd7440d99455aa0134d3e7e25c8cb, State: Running, Role: LEADER
I20240501 20:01:33.096580 28958 consensus_queue.cc:237] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2fd7440d99455aa0134d3e7e25c8cb" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 32851 } } peers { permanent_uuid: "7249ca819fd14f9e92070d573add724a" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 42719 } } peers { permanent_uuid: "b81faa4c3db94ac6a8bc782a5a95d2e2" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 45213 } }
I20240501 20:01:33.103025 28655 catalog_manager.cc:5484] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f2fd7440d99455aa0134d3e7e25c8cb (127.24.69.130). New cstate: current_term: 1 leader_uuid: "8f2fd7440d99455aa0134d3e7e25c8cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2fd7440d99455aa0134d3e7e25c8cb" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 32851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7249ca819fd14f9e92070d573add724a" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b81faa4c3db94ac6a8bc782a5a95d2e2" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 45213 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:33.149554 24854 tablet_server.cc:178] TabletServer@127.24.69.129:0 shutting down...
I20240501 20:01:33.164654 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:01:33.165555 24854 tablet_replica.cc:330] T 0ca4d2159b1548a880e9ea07077a8969 P 7249ca819fd14f9e92070d573add724a: stopping tablet replica
I20240501 20:01:33.166045 24854 raft_consensus.cc:2229] T 0ca4d2159b1548a880e9ea07077a8969 P 7249ca819fd14f9e92070d573add724a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:33.166419 24854 raft_consensus.cc:2258] T 0ca4d2159b1548a880e9ea07077a8969 P 7249ca819fd14f9e92070d573add724a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:33.184410 24854 tablet_server.cc:194] TabletServer@127.24.69.129:0 shutdown complete.
I20240501 20:01:33.456990 28709 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:33.458199 28709 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:01:33.458492 28709 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240501 20:01:33.542119 28797 consensus_peers.cc:485] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb -> Peer 7249ca819fd14f9e92070d573add724a (127.24.69.129:42719): Couldn't send request to peer 7249ca819fd14f9e92070d573add724a. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:42719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:33.598024 28958 consensus_queue.cc:1035] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81faa4c3db94ac6a8bc782a5a95d2e2" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 45213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:01:34.558032 28797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.69.129:42719: connect: Connection refused (error 111) [suppressed 17 similar messages]
I20240501 20:01:35.459396 28709 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:35.460734 28709 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:01:35.461079 28709 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240501 20:01:36.014829 28797 consensus_peers.cc:485] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb -> Peer 7249ca819fd14f9e92070d573add724a (127.24.69.129:42719): Couldn't send request to peer 7249ca819fd14f9e92070d573add724a. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:42719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:01:37.462473 28709 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:37.464380 28709 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:01:37.464671 28709 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240501 20:01:38.476639 28797 consensus_peers.cc:485] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb -> Peer 7249ca819fd14f9e92070d573add724a (127.24.69.129:42719): Couldn't send request to peer 7249ca819fd14f9e92070d573add724a. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:42719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:01:39.465528 28709 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:39.466809 28709 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:01:39.467080 28709 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240501 20:01:39.881379 28797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.69.129:42719: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 20:01:40.787194 28797 consensus_peers.cc:485] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb -> Peer 7249ca819fd14f9e92070d573add724a (127.24.69.129:42719): Couldn't send request to peer 7249ca819fd14f9e92070d573add724a. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:42719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:01:41.467921 28709 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:41.469194 28709 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:01:41.469486 28709 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240501 20:01:43.219808 28797 consensus_peers.cc:485] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb -> Peer 7249ca819fd14f9e92070d573add724a (127.24.69.129:42719): Couldn't send request to peer 7249ca819fd14f9e92070d573add724a. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:42719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:01:43.470353 28709 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:43.471614 28709 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:01:43.471911 28709 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:01:44.253082 24854 tablet_server.cc:178] TabletServer@127.24.69.130:0 shutting down...
I20240501 20:01:44.277480 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:01:44.278182 24854 tablet_replica.cc:330] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb: stopping tablet replica
I20240501 20:01:44.278820 24854 raft_consensus.cc:2229] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:01:44.279798 24854 raft_consensus.cc:2258] T 0ca4d2159b1548a880e9ea07077a8969 P 8f2fd7440d99455aa0134d3e7e25c8cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:44.300443 24854 tablet_server.cc:194] TabletServer@127.24.69.130:0 shutdown complete.
I20240501 20:01:44.312057 24854 tablet_server.cc:178] TabletServer@127.24.69.131:0 shutting down...
I20240501 20:01:44.331449 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:01:44.332131 24854 tablet_replica.cc:330] T 0ca4d2159b1548a880e9ea07077a8969 P b81faa4c3db94ac6a8bc782a5a95d2e2: stopping tablet replica
I20240501 20:01:44.332733 24854 raft_consensus.cc:2229] T 0ca4d2159b1548a880e9ea07077a8969 P b81faa4c3db94ac6a8bc782a5a95d2e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:44.333192 24854 raft_consensus.cc:2258] T 0ca4d2159b1548a880e9ea07077a8969 P b81faa4c3db94ac6a8bc782a5a95d2e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:44.352394 24854 tablet_server.cc:194] TabletServer@127.24.69.131:0 shutdown complete.
I20240501 20:01:44.362077 24854 master.cc:537] Master@127.24.69.190:36253 shutting down...
I20240501 20:01:44.379932 24854 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 07f966c955ab4f899f50206b048f68b2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:01:44.380599 24854 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 07f966c955ab4f899f50206b048f68b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:44.380954 24854 tablet_replica.cc:330] T 00000000000000000000000000000000 P 07f966c955ab4f899f50206b048f68b2: stopping tablet replica
I20240501 20:01:44.398770 24854 master.cc:552] Master@127.24.69.190:36253 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor (13121 ms)
[ RUN ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering
I20240501 20:01:44.433763 24854 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.69.190:44515
I20240501 20:01:44.434770 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:44.440687 28990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:44.441102 28991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:44.442073 28993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:44.442960 24854 server_base.cc:1034] running on GCE node
I20240501 20:01:44.443660 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:01:44.443840 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:44.443979 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593704443964 us; error 0 us; skew 500 ppm
I20240501 20:01:44.444543 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:44.452935 24854 webserver.cc:460] Webserver started at http://127.24.69.190:40135/ using document root <none> and password file <none>
I20240501 20:01:44.453392 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:44.453552 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:44.453804 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:44.454996 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714593576104309-24854-0/minicluster-data/master-0-root/instance:
uuid: "abb684196a634ebd9065a952ce95407c"
format_stamp: "Formatted at 2024-05-01 20:01:44 on dist-test-slave-z8x3"
I20240501 20:01:44.459208 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 20:01:44.462311 28998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:44.462966 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:01:44.463215 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714593576104309-24854-0/minicluster-data/master-0-root
uuid: "abb684196a634ebd9065a952ce95407c"
format_stamp: "Formatted at 2024-05-01 20:01:44 on dist-test-slave-z8x3"
I20240501 20:01:44.463444 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714593576104309-24854-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714593576104309-24854-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714593576104309-24854-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:44.475979 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:44.477056 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:44.512727 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.190:44515
I20240501 20:01:44.512809 29049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.190:44515 every 8 connection(s)
I20240501 20:01:44.516520 29050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:44.527370 29050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abb684196a634ebd9065a952ce95407c: Bootstrap starting.
I20240501 20:01:44.531361 29050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abb684196a634ebd9065a952ce95407c: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:44.535115 29050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abb684196a634ebd9065a952ce95407c: No bootstrap required, opened a new log
I20240501 20:01:44.537031 29050 raft_consensus.cc:348] T 00000000000000000000000000000000 P abb684196a634ebd9065a952ce95407c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb684196a634ebd9065a952ce95407c" member_type: VOTER }
I20240501 20:01:44.537483 29050 raft_consensus.cc:374] T 00000000000000000000000000000000 P abb684196a634ebd9065a952ce95407c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:44.537663 29050 raft_consensus.cc:729] T 00000000000000000000000000000000 P abb684196a634ebd9065a952ce95407c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb684196a634ebd9065a952ce95407c, State: Initialized, Role: FOLLOWER
I20240501 20:01:44.538170 29050 consensus_queue.cc:260] T 00000000000000000000000000000000 P abb684196a634ebd9065a952ce95407c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb684196a634ebd9065a952ce95407c" member_type: VOTER }
I20240501 20:01:44.538604 29050 raft_consensus.cc:388] T 00000000000000000000000000000000 P abb684196a634ebd9065a952ce95407c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:44.538790 29050 raft_consensus.cc:482] T 00000000000000000000000000000000 P abb684196a634ebd9065a952ce95407c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:44.539003 29050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abb684196a634ebd9065a952ce95407c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:44.543195 29050 raft_consensus.cc:504] T 00000000000000000000000000000000 P abb684196a634ebd9065a952ce95407c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb684196a634ebd9065a952ce95407c" member_type: VOTER }
I20240501 20:01:44.543653 29050 leader_election.cc:302] T 00000000000000000000000000000000 P abb684196a634ebd9065a952ce95407c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abb684196a634ebd9065a952ce95407c; no voters:
I20240501 20:01:44.544682 29050 leader_election.cc:288] T 00000000000000000000000000000000 P abb684196a634ebd9065a952ce95407c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:44.545070 29053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abb684196a634ebd9065a952ce95407c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:44.546360 29053 raft_consensus.cc:686] T 00000000000000000000000000000000 P abb684196a634ebd9065a952ce95407c [term 1 LEADER]: Becoming Leader. State: Replica: abb684196a634ebd9065a952ce95407c, State: Running, Role: LEADER
I20240501 20:01:44.546955 29053 consensus_queue.cc:237] T 00000000000000000000000000000000 P abb684196a634ebd9065a952ce95407c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb684196a634ebd9065a952ce95407c" member_type: VOTER }
I20240501 20:01:44.547474 29050 sys_catalog.cc:564] T 00000000000000000000000000000000 P abb684196a634ebd9065a952ce95407c [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:44.549988 29054 sys_catalog.cc:455] T 00000000000000000000000000000000 P abb684196a634ebd9065a952ce95407c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abb684196a634ebd9065a952ce95407c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb684196a634ebd9065a952ce95407c" member_type: VOTER } }
I20240501 20:01:44.550091 29055 sys_catalog.cc:455] T 00000000000000000000000000000000 P abb684196a634ebd9065a952ce95407c [sys.catalog]: SysCatalogTable state changed. Reason: New leader abb684196a634ebd9065a952ce95407c. Latest consensus state: current_term: 1 leader_uuid: "abb684196a634ebd9065a952ce95407c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb684196a634ebd9065a952ce95407c" member_type: VOTER } }
I20240501 20:01:44.550819 29055 sys_catalog.cc:458] T 00000000000000000000000000000000 P abb684196a634ebd9065a952ce95407c [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:44.550760 29054 sys_catalog.cc:458] T 00000000000000000000000000000000 P abb684196a634ebd9065a952ce95407c [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:44.557364 29061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:44.563015 29061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:44.563967 24854 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 20:01:44.571065 29061 catalog_manager.cc:1347] Generated new cluster ID: 269793ebad784aa3ac226745fecca50a
I20240501 20:01:44.571333 29061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:44.613617 29061 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:44.614876 29061 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:44.636076 29061 catalog_manager.cc:5857] T 00000000000000000000000000000000 P abb684196a634ebd9065a952ce95407c: Generated new TSK 0
I20240501 20:01:44.636708 29061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:01:44.694746 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:44.701488 29071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:44.702054 29072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:44.703146 29074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:44.703902 24854 server_base.cc:1034] running on GCE node
I20240501 20:01:44.704644 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:01:44.704825 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:44.704962 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593704704951 us; error 0 us; skew 500 ppm
I20240501 20:01:44.705407 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:44.707595 24854 webserver.cc:460] Webserver started at http://127.24.69.129:44661/ using document root <none> and password file <none>
I20240501 20:01:44.708006 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:44.708158 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:44.708407 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:44.709443 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714593576104309-24854-0/minicluster-data/ts-0-root/instance:
uuid: "a1a42f9b0ed84cc08e0819f874e79db9"
format_stamp: "Formatted at 2024-05-01 20:01:44 on dist-test-slave-z8x3"
I20240501 20:01:44.713610 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 20:01:44.716626 29079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:44.717343 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:01:44.717567 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714593576104309-24854-0/minicluster-data/ts-0-root
uuid: "a1a42f9b0ed84cc08e0819f874e79db9"
format_stamp: "Formatted at 2024-05-01 20:01:44 on dist-test-slave-z8x3"
I20240501 20:01:44.717795 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714593576104309-24854-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714593576104309-24854-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714593576104309-24854-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:44.746862 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:44.748008 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:44.749431 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:44.751658 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:44.751849 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:44.752075 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:44.752239 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:44.789990 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.129:44987
I20240501 20:01:44.790071 29141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.129:44987 every 8 connection(s)
I20240501 20:01:44.794406 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:44.802238 29146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:44.803265 29147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:44.804507 29149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:44.805378 24854 server_base.cc:1034] running on GCE node
I20240501 20:01:44.806636 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240501 20:01:44.806761 29142 heartbeater.cc:347] Connected to a master server at 127.24.69.190:44515
W20240501 20:01:44.806922 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:44.807144 29142 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:44.807132 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593704807114 us; error 0 us; skew 500 ppm
I20240501 20:01:44.808080 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:44.808136 29142 heartbeater.cc:511] Master 127.24.69.190:44515 requested a full tablet report, sending...
I20240501 20:01:44.810328 29015 ts_manager.cc:194] Registered new tserver with Master: a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987)
I20240501 20:01:44.810850 24854 webserver.cc:460] Webserver started at http://127.24.69.130:35413/ using document root <none> and password file <none>
I20240501 20:01:44.811326 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:44.811511 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:44.811743 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:44.812170 29015 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45470
I20240501 20:01:44.812959 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714593576104309-24854-0/minicluster-data/ts-1-root/instance:
uuid: "bf3f337ceb234e2fbf26e507c009a87f"
format_stamp: "Formatted at 2024-05-01 20:01:44 on dist-test-slave-z8x3"
I20240501 20:01:44.817193 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20240501 20:01:44.820504 29154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:44.821229 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 20:01:44.821497 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714593576104309-24854-0/minicluster-data/ts-1-root
uuid: "bf3f337ceb234e2fbf26e507c009a87f"
format_stamp: "Formatted at 2024-05-01 20:01:44 on dist-test-slave-z8x3"
I20240501 20:01:44.821767 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714593576104309-24854-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714593576104309-24854-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714593576104309-24854-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:44.864725 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:44.865859 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:44.867198 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:44.869336 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:44.869531 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:44.869755 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:44.869904 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:44.906932 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.130:42795
I20240501 20:01:44.907037 29216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.130:42795 every 8 connection(s)
I20240501 20:01:44.911581 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:44.917923 29220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:44.918939 29221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:44.921258 29223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:44.921970 24854 server_base.cc:1034] running on GCE node
I20240501 20:01:44.922497 29217 heartbeater.cc:347] Connected to a master server at 127.24.69.190:44515
I20240501 20:01:44.922849 29217 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:44.922924 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:01:44.923208 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:44.923398 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593704923376 us; error 0 us; skew 500 ppm
I20240501 20:01:44.923605 29217 heartbeater.cc:511] Master 127.24.69.190:44515 requested a full tablet report, sending...
I20240501 20:01:44.924031 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:44.925442 29015 ts_manager.cc:194] Registered new tserver with Master: bf3f337ceb234e2fbf26e507c009a87f (127.24.69.130:42795)
I20240501 20:01:44.926715 29015 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45480
I20240501 20:01:44.928818 24854 webserver.cc:460] Webserver started at http://127.24.69.131:42659/ using document root <none> and password file <none>
I20240501 20:01:44.929279 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:44.929458 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:44.929689 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:44.930800 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714593576104309-24854-0/minicluster-data/ts-2-root/instance:
uuid: "6621104ea62e411da47ab612f9f26e89"
format_stamp: "Formatted at 2024-05-01 20:01:44 on dist-test-slave-z8x3"
I20240501 20:01:44.934860 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 20:01:44.937953 29228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:44.938695 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:01:44.938978 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714593576104309-24854-0/minicluster-data/ts-2-root
uuid: "6621104ea62e411da47ab612f9f26e89"
format_stamp: "Formatted at 2024-05-01 20:01:44 on dist-test-slave-z8x3"
I20240501 20:01:44.939258 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714593576104309-24854-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714593576104309-24854-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714593576104309-24854-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:45.004918 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:45.006026 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:45.007242 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:45.009402 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:45.009603 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:45.009825 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:45.009975 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:45.049149 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.131:37289
I20240501 20:01:45.049286 29290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.131:37289 every 8 connection(s)
I20240501 20:01:45.062266 29291 heartbeater.cc:347] Connected to a master server at 127.24.69.190:44515
I20240501 20:01:45.062616 29291 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:45.063256 29291 heartbeater.cc:511] Master 127.24.69.190:44515 requested a full tablet report, sending...
I20240501 20:01:45.065110 29015 ts_manager.cc:194] Registered new tserver with Master: 6621104ea62e411da47ab612f9f26e89 (127.24.69.131:37289)
I20240501 20:01:45.065704 24854 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013512417s
I20240501 20:01:45.066458 29015 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45488
I20240501 20:01:45.584494 24854 test_util.cc:268] Using random seed: -188651693
I20240501 20:01:45.605751 29015 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45500:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 20:01:45.607969 29015 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:01:45.668207 29256 tablet_service.cc:1467] Processing CreateTablet for tablet 82aaf53585ea4d87aa1d65dc40658d3f (DEFAULT_TABLE table=test-workload [id=bff1774076514b6b9b88f12033448b1e]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240501 20:01:45.669008 29107 tablet_service.cc:1467] Processing CreateTablet for tablet 82aaf53585ea4d87aa1d65dc40658d3f (DEFAULT_TABLE table=test-workload [id=bff1774076514b6b9b88f12033448b1e]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240501 20:01:45.669634 29256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82aaf53585ea4d87aa1d65dc40658d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:45.670293 29107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82aaf53585ea4d87aa1d65dc40658d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:45.673416 29255 tablet_service.cc:1467] Processing CreateTablet for tablet 128e1005f1df4010982d9ebf0c02f727 (DEFAULT_TABLE table=test-workload [id=bff1774076514b6b9b88f12033448b1e]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240501 20:01:45.674783 29255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128e1005f1df4010982d9ebf0c02f727. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:45.677822 29106 tablet_service.cc:1467] Processing CreateTablet for tablet 128e1005f1df4010982d9ebf0c02f727 (DEFAULT_TABLE table=test-workload [id=bff1774076514b6b9b88f12033448b1e]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240501 20:01:45.678216 29254 tablet_service.cc:1467] Processing CreateTablet for tablet 27b9198949354d2e9202677a25193fa9 (DEFAULT_TABLE table=test-workload [id=bff1774076514b6b9b88f12033448b1e]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240501 20:01:45.679167 29106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128e1005f1df4010982d9ebf0c02f727. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:45.679539 29254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b9198949354d2e9202677a25193fa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:45.681480 29253 tablet_service.cc:1467] Processing CreateTablet for tablet 2587cb164a134d5f9112e11a7176e223 (DEFAULT_TABLE table=test-workload [id=bff1774076514b6b9b88f12033448b1e]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240501 20:01:45.681617 29182 tablet_service.cc:1467] Processing CreateTablet for tablet 82aaf53585ea4d87aa1d65dc40658d3f (DEFAULT_TABLE table=test-workload [id=bff1774076514b6b9b88f12033448b1e]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240501 20:01:45.682279 29179 tablet_service.cc:1467] Processing CreateTablet for tablet 2587cb164a134d5f9112e11a7176e223 (DEFAULT_TABLE table=test-workload [id=bff1774076514b6b9b88f12033448b1e]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240501 20:01:45.682760 29253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2587cb164a134d5f9112e11a7176e223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:45.682929 29182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82aaf53585ea4d87aa1d65dc40658d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:45.684464 29181 tablet_service.cc:1467] Processing CreateTablet for tablet 128e1005f1df4010982d9ebf0c02f727 (DEFAULT_TABLE table=test-workload [id=bff1774076514b6b9b88f12033448b1e]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240501 20:01:45.685770 29181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128e1005f1df4010982d9ebf0c02f727. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:45.688395 29105 tablet_service.cc:1467] Processing CreateTablet for tablet 27b9198949354d2e9202677a25193fa9 (DEFAULT_TABLE table=test-workload [id=bff1774076514b6b9b88f12033448b1e]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240501 20:01:45.689626 29105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b9198949354d2e9202677a25193fa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:45.691453 29180 tablet_service.cc:1467] Processing CreateTablet for tablet 27b9198949354d2e9202677a25193fa9 (DEFAULT_TABLE table=test-workload [id=bff1774076514b6b9b88f12033448b1e]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240501 20:01:45.692657 29180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b9198949354d2e9202677a25193fa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:45.693990 29104 tablet_service.cc:1467] Processing CreateTablet for tablet 2587cb164a134d5f9112e11a7176e223 (DEFAULT_TABLE table=test-workload [id=bff1774076514b6b9b88f12033448b1e]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240501 20:01:45.695175 29104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2587cb164a134d5f9112e11a7176e223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:45.696589 29179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2587cb164a134d5f9112e11a7176e223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:45.731778 29310 tablet_bootstrap.cc:492] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89: Bootstrap starting.
I20240501 20:01:45.743897 29311 tablet_bootstrap.cc:492] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9: Bootstrap starting.
I20240501 20:01:45.749303 29312 tablet_bootstrap.cc:492] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f: Bootstrap starting.
I20240501 20:01:45.749387 29311 tablet_bootstrap.cc:654] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:45.750802 29310 tablet_bootstrap.cc:654] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:45.756526 29312 tablet_bootstrap.cc:654] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:45.757565 29311 tablet_bootstrap.cc:492] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9: No bootstrap required, opened a new log
I20240501 20:01:45.758093 29310 tablet_bootstrap.cc:492] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89: No bootstrap required, opened a new log
I20240501 20:01:45.758062 29311 ts_tablet_manager.cc:1367] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.001s
I20240501 20:01:45.758663 29310 ts_tablet_manager.cc:1367] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.003s
I20240501 20:01:45.760841 29310 raft_consensus.cc:348] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:45.761570 29310 raft_consensus.cc:374] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:45.761178 29311 raft_consensus.cc:348] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:45.761926 29310 raft_consensus.cc:729] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6621104ea62e411da47ab612f9f26e89, State: Initialized, Role: FOLLOWER
I20240501 20:01:45.762087 29311 raft_consensus.cc:374] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:45.762524 29311 raft_consensus.cc:729] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a42f9b0ed84cc08e0819f874e79db9, State: Initialized, Role: FOLLOWER
I20240501 20:01:45.762712 29312 tablet_bootstrap.cc:492] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f: No bootstrap required, opened a new log
I20240501 20:01:45.763401 29312 ts_tablet_manager.cc:1367] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20240501 20:01:45.763141 29310 consensus_queue.cc:260] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:45.763396 29311 consensus_queue.cc:260] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:45.766072 29291 heartbeater.cc:503] Master 127.24.69.190:44515 was elected leader, sending a full tablet report...
I20240501 20:01:45.766904 29312 raft_consensus.cc:348] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:45.767975 29312 raft_consensus.cc:374] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:45.768441 29310 ts_tablet_manager.cc:1398] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20240501 20:01:45.768441 29312 raft_consensus.cc:729] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf3f337ceb234e2fbf26e507c009a87f, State: Initialized, Role: FOLLOWER
I20240501 20:01:45.769603 29310 tablet_bootstrap.cc:492] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89: Bootstrap starting.
I20240501 20:01:45.769858 29312 consensus_queue.cc:260] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:45.773375 29311 ts_tablet_manager.cc:1398] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9: Time spent starting tablet: real 0.015s user 0.007s sys 0.005s
I20240501 20:01:45.774302 29142 heartbeater.cc:503] Master 127.24.69.190:44515 was elected leader, sending a full tablet report...
I20240501 20:01:45.775758 29310 tablet_bootstrap.cc:654] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:45.777809 29311 tablet_bootstrap.cc:492] T 82aaf53585ea4d87aa1d65dc40658d3f P a1a42f9b0ed84cc08e0819f874e79db9: Bootstrap starting.
I20240501 20:01:45.779238 29312 ts_tablet_manager.cc:1398] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20240501 20:01:45.780081 29217 heartbeater.cc:503] Master 127.24.69.190:44515 was elected leader, sending a full tablet report...
I20240501 20:01:45.786339 29317 raft_consensus.cc:482] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:45.788203 29312 tablet_bootstrap.cc:492] T 82aaf53585ea4d87aa1d65dc40658d3f P bf3f337ceb234e2fbf26e507c009a87f: Bootstrap starting.
I20240501 20:01:45.789873 29317 raft_consensus.cc:504] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:45.795447 29312 tablet_bootstrap.cc:654] T 82aaf53585ea4d87aa1d65dc40658d3f P bf3f337ceb234e2fbf26e507c009a87f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:45.800874 29310 tablet_bootstrap.cc:492] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89: No bootstrap required, opened a new log
I20240501 20:01:45.801421 29310 ts_tablet_manager.cc:1367] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.006s
I20240501 20:01:45.804121 29312 tablet_bootstrap.cc:492] T 82aaf53585ea4d87aa1d65dc40658d3f P bf3f337ceb234e2fbf26e507c009a87f: No bootstrap required, opened a new log
I20240501 20:01:45.804314 29310 raft_consensus.cc:348] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } }
I20240501 20:01:45.804852 29312 ts_tablet_manager.cc:1367] T 82aaf53585ea4d87aa1d65dc40658d3f P bf3f337ceb234e2fbf26e507c009a87f: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.004s
I20240501 20:01:45.805387 29310 raft_consensus.cc:374] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:45.805780 29310 raft_consensus.cc:729] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6621104ea62e411da47ab612f9f26e89, State: Initialized, Role: FOLLOWER
I20240501 20:01:45.806393 29310 consensus_queue.cc:260] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } }
I20240501 20:01:45.808302 29310 ts_tablet_manager.cc:1398] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20240501 20:01:45.809245 29310 tablet_bootstrap.cc:492] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89: Bootstrap starting.
I20240501 20:01:45.814265 29312 raft_consensus.cc:348] T 82aaf53585ea4d87aa1d65dc40658d3f P bf3f337ceb234e2fbf26e507c009a87f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } }
I20240501 20:01:45.815362 29310 tablet_bootstrap.cc:654] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:45.816426 29317 leader_election.cc:288] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf3f337ceb234e2fbf26e507c009a87f (127.24.69.130:42795), 6621104ea62e411da47ab612f9f26e89 (127.24.69.131:37289)
I20240501 20:01:45.815264 29312 raft_consensus.cc:374] T 82aaf53585ea4d87aa1d65dc40658d3f P bf3f337ceb234e2fbf26e507c009a87f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:45.818687 29312 raft_consensus.cc:729] T 82aaf53585ea4d87aa1d65dc40658d3f P bf3f337ceb234e2fbf26e507c009a87f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf3f337ceb234e2fbf26e507c009a87f, State: Initialized, Role: FOLLOWER
I20240501 20:01:45.819571 29312 consensus_queue.cc:260] T 82aaf53585ea4d87aa1d65dc40658d3f P bf3f337ceb234e2fbf26e507c009a87f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } }
I20240501 20:01:45.821841 29312 ts_tablet_manager.cc:1398] T 82aaf53585ea4d87aa1d65dc40658d3f P bf3f337ceb234e2fbf26e507c009a87f: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20240501 20:01:45.824484 29312 tablet_bootstrap.cc:492] T 128e1005f1df4010982d9ebf0c02f727 P bf3f337ceb234e2fbf26e507c009a87f: Bootstrap starting.
I20240501 20:01:45.832181 29311 tablet_bootstrap.cc:654] T 82aaf53585ea4d87aa1d65dc40658d3f P a1a42f9b0ed84cc08e0819f874e79db9: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:45.839066 29312 tablet_bootstrap.cc:654] T 128e1005f1df4010982d9ebf0c02f727 P bf3f337ceb234e2fbf26e507c009a87f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:45.844725 29311 tablet_bootstrap.cc:492] T 82aaf53585ea4d87aa1d65dc40658d3f P a1a42f9b0ed84cc08e0819f874e79db9: No bootstrap required, opened a new log
I20240501 20:01:45.845269 29311 ts_tablet_manager.cc:1367] T 82aaf53585ea4d87aa1d65dc40658d3f P a1a42f9b0ed84cc08e0819f874e79db9: Time spent bootstrapping tablet: real 0.068s user 0.015s sys 0.003s
I20240501 20:01:45.847153 29192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128e1005f1df4010982d9ebf0c02f727" candidate_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf3f337ceb234e2fbf26e507c009a87f" is_pre_election: true
W20240501 20:01:45.848671 29081 leader_election.cc:341] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf3f337ceb234e2fbf26e507c009a87f (127.24.69.130:42795): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:45.850618 29311 raft_consensus.cc:348] T 82aaf53585ea4d87aa1d65dc40658d3f P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } }
I20240501 20:01:45.851449 29312 tablet_bootstrap.cc:492] T 128e1005f1df4010982d9ebf0c02f727 P bf3f337ceb234e2fbf26e507c009a87f: No bootstrap required, opened a new log
I20240501 20:01:45.851402 29311 raft_consensus.cc:374] T 82aaf53585ea4d87aa1d65dc40658d3f P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:45.852312 29311 raft_consensus.cc:729] T 82aaf53585ea4d87aa1d65dc40658d3f P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a42f9b0ed84cc08e0819f874e79db9, State: Initialized, Role: FOLLOWER
I20240501 20:01:45.852296 29266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128e1005f1df4010982d9ebf0c02f727" candidate_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6621104ea62e411da47ab612f9f26e89" is_pre_election: true
I20240501 20:01:45.853183 29266 raft_consensus.cc:2454] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1a42f9b0ed84cc08e0819f874e79db9 in term 0.
I20240501 20:01:45.853174 29311 consensus_queue.cc:260] T 82aaf53585ea4d87aa1d65dc40658d3f P a1a42f9b0ed84cc08e0819f874e79db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } }
I20240501 20:01:45.853946 29310 tablet_bootstrap.cc:492] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89: No bootstrap required, opened a new log
I20240501 20:01:45.854470 29310 ts_tablet_manager.cc:1367] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89: Time spent bootstrapping tablet: real 0.045s user 0.013s sys 0.011s
I20240501 20:01:45.854720 29080 leader_election.cc:302] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6621104ea62e411da47ab612f9f26e89, a1a42f9b0ed84cc08e0819f874e79db9; no voters: bf3f337ceb234e2fbf26e507c009a87f
I20240501 20:01:45.855504 29311 ts_tablet_manager.cc:1398] T 82aaf53585ea4d87aa1d65dc40658d3f P a1a42f9b0ed84cc08e0819f874e79db9: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240501 20:01:45.852392 29312 ts_tablet_manager.cc:1367] T 128e1005f1df4010982d9ebf0c02f727 P bf3f337ceb234e2fbf26e507c009a87f: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.005s
I20240501 20:01:45.855948 29319 raft_consensus.cc:2781] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:45.856338 29319 raft_consensus.cc:482] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:45.856530 29311 tablet_bootstrap.cc:492] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9: Bootstrap starting.
I20240501 20:01:45.856658 29319 raft_consensus.cc:3037] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:45.857357 29310 raft_consensus.cc:348] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:45.858067 29310 raft_consensus.cc:374] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:45.858359 29310 raft_consensus.cc:729] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6621104ea62e411da47ab612f9f26e89, State: Initialized, Role: FOLLOWER
I20240501 20:01:45.858310 29312 raft_consensus.cc:348] T 128e1005f1df4010982d9ebf0c02f727 P bf3f337ceb234e2fbf26e507c009a87f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:45.859169 29312 raft_consensus.cc:374] T 128e1005f1df4010982d9ebf0c02f727 P bf3f337ceb234e2fbf26e507c009a87f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:45.859483 29312 raft_consensus.cc:729] T 128e1005f1df4010982d9ebf0c02f727 P bf3f337ceb234e2fbf26e507c009a87f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf3f337ceb234e2fbf26e507c009a87f, State: Initialized, Role: FOLLOWER
I20240501 20:01:45.859274 29310 consensus_queue.cc:260] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:45.862470 29311 tablet_bootstrap.cc:654] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:45.860304 29312 consensus_queue.cc:260] T 128e1005f1df4010982d9ebf0c02f727 P bf3f337ceb234e2fbf26e507c009a87f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:45.865574 29312 ts_tablet_manager.cc:1398] T 128e1005f1df4010982d9ebf0c02f727 P bf3f337ceb234e2fbf26e507c009a87f: Time spent starting tablet: real 0.010s user 0.004s sys 0.003s
I20240501 20:01:45.866494 29312 tablet_bootstrap.cc:492] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f: Bootstrap starting.
I20240501 20:01:45.867399 29316 raft_consensus.cc:482] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:45.867839 29316 raft_consensus.cc:504] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:45.872154 29312 tablet_bootstrap.cc:654] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:45.885929 29319 raft_consensus.cc:504] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:45.886386 29316 leader_election.cc:288] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987), bf3f337ceb234e2fbf26e507c009a87f (127.24.69.130:42795)
I20240501 20:01:45.888701 29319 leader_election.cc:288] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9 [CANDIDATE]: Term 1 election: Requested vote from peers bf3f337ceb234e2fbf26e507c009a87f (127.24.69.130:42795), 6621104ea62e411da47ab612f9f26e89 (127.24.69.131:37289)
I20240501 20:01:45.889418 29192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128e1005f1df4010982d9ebf0c02f727" candidate_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf3f337ceb234e2fbf26e507c009a87f"
I20240501 20:01:45.889701 29266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128e1005f1df4010982d9ebf0c02f727" candidate_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6621104ea62e411da47ab612f9f26e89"
I20240501 20:01:45.890027 29192 raft_consensus.cc:3037] T 128e1005f1df4010982d9ebf0c02f727 P bf3f337ceb234e2fbf26e507c009a87f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:45.890242 29266 raft_consensus.cc:3037] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:45.893687 29311 tablet_bootstrap.cc:492] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9: No bootstrap required, opened a new log
I20240501 20:01:45.894112 29311 ts_tablet_manager.cc:1367] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9: Time spent bootstrapping tablet: real 0.038s user 0.018s sys 0.018s
I20240501 20:01:45.895995 29311 raft_consensus.cc:348] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:45.896886 29311 raft_consensus.cc:374] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:45.897274 29266 raft_consensus.cc:2454] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1a42f9b0ed84cc08e0819f874e79db9 in term 1.
I20240501 20:01:45.897322 29311 raft_consensus.cc:729] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a42f9b0ed84cc08e0819f874e79db9, State: Initialized, Role: FOLLOWER
I20240501 20:01:45.898636 29080 leader_election.cc:302] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6621104ea62e411da47ab612f9f26e89, a1a42f9b0ed84cc08e0819f874e79db9; no voters:
I20240501 20:01:45.899495 29319 raft_consensus.cc:2781] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:45.898793 29311 consensus_queue.cc:260] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:45.899986 29319 raft_consensus.cc:686] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9 [term 1 LEADER]: Becoming Leader. State: Replica: a1a42f9b0ed84cc08e0819f874e79db9, State: Running, Role: LEADER
I20240501 20:01:45.900797 29319 consensus_queue.cc:237] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:45.903093 29311 ts_tablet_manager.cc:1398] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20240501 20:01:45.904420 29311 tablet_bootstrap.cc:492] T 2587cb164a134d5f9112e11a7176e223 P a1a42f9b0ed84cc08e0819f874e79db9: Bootstrap starting.
I20240501 20:01:45.905548 29192 raft_consensus.cc:2454] T 128e1005f1df4010982d9ebf0c02f727 P bf3f337ceb234e2fbf26e507c009a87f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1a42f9b0ed84cc08e0819f874e79db9 in term 1.
I20240501 20:01:45.911651 29311 tablet_bootstrap.cc:654] T 2587cb164a134d5f9112e11a7176e223 P a1a42f9b0ed84cc08e0819f874e79db9: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:45.914561 29310 ts_tablet_manager.cc:1398] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89: Time spent starting tablet: real 0.060s user 0.001s sys 0.004s
I20240501 20:01:45.926523 29310 tablet_bootstrap.cc:492] T 2587cb164a134d5f9112e11a7176e223 P 6621104ea62e411da47ab612f9f26e89: Bootstrap starting.
I20240501 20:01:45.932147 29117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b9198949354d2e9202677a25193fa9" candidate_uuid: "6621104ea62e411da47ab612f9f26e89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" is_pre_election: true
I20240501 20:01:45.937238 29317 raft_consensus.cc:482] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:45.937943 29317 raft_consensus.cc:504] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:45.940279 29013 catalog_manager.cc:5484] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129). New cstate: current_term: 1 leader_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:45.942101 29117 raft_consensus.cc:2454] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6621104ea62e411da47ab612f9f26e89 in term 0.
I20240501 20:01:45.942975 29191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b9198949354d2e9202677a25193fa9" candidate_uuid: "6621104ea62e411da47ab612f9f26e89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf3f337ceb234e2fbf26e507c009a87f" is_pre_election: true
I20240501 20:01:45.943701 29191 raft_consensus.cc:2454] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6621104ea62e411da47ab612f9f26e89 in term 0.
I20240501 20:01:45.944134 29266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b9198949354d2e9202677a25193fa9" candidate_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6621104ea62e411da47ab612f9f26e89" is_pre_election: true
I20240501 20:01:45.944752 29266 raft_consensus.cc:2454] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1a42f9b0ed84cc08e0819f874e79db9 in term 0.
I20240501 20:01:45.945082 29230 leader_election.cc:302] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6621104ea62e411da47ab612f9f26e89, bf3f337ceb234e2fbf26e507c009a87f; no voters:
I20240501 20:01:45.945930 29316 raft_consensus.cc:2781] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:45.945888 29080 leader_election.cc:302] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6621104ea62e411da47ab612f9f26e89, a1a42f9b0ed84cc08e0819f874e79db9; no voters:
I20240501 20:01:45.946390 29316 raft_consensus.cc:482] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:45.946734 29316 raft_consensus.cc:3037] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:45.949905 29317 leader_election.cc:288] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf3f337ceb234e2fbf26e507c009a87f (127.24.69.130:42795), 6621104ea62e411da47ab612f9f26e89 (127.24.69.131:37289)
I20240501 20:01:45.950685 29317 raft_consensus.cc:2781] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:45.952306 29317 raft_consensus.cc:482] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:45.952860 29317 raft_consensus.cc:3037] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:45.951522 29192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b9198949354d2e9202677a25193fa9" candidate_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf3f337ceb234e2fbf26e507c009a87f" is_pre_election: true
I20240501 20:01:45.955641 29192 raft_consensus.cc:2454] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1a42f9b0ed84cc08e0819f874e79db9 in term 0.
I20240501 20:01:45.964812 29317 raft_consensus.cc:504] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:45.969302 29316 raft_consensus.cc:504] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:45.973302 29117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b9198949354d2e9202677a25193fa9" candidate_uuid: "6621104ea62e411da47ab612f9f26e89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a42f9b0ed84cc08e0819f874e79db9"
I20240501 20:01:45.974062 29117 raft_consensus.cc:2379] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6621104ea62e411da47ab612f9f26e89 in current term 1: Already voted for candidate a1a42f9b0ed84cc08e0819f874e79db9 in this term.
I20240501 20:01:45.975009 29192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b9198949354d2e9202677a25193fa9" candidate_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf3f337ceb234e2fbf26e507c009a87f"
I20240501 20:01:45.975872 29192 raft_consensus.cc:3037] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:45.976660 29191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b9198949354d2e9202677a25193fa9" candidate_uuid: "6621104ea62e411da47ab612f9f26e89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf3f337ceb234e2fbf26e507c009a87f"
I20240501 20:01:45.978729 29230 leader_election.cc:302] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6621104ea62e411da47ab612f9f26e89; no voters: a1a42f9b0ed84cc08e0819f874e79db9, bf3f337ceb234e2fbf26e507c009a87f
I20240501 20:01:45.984989 29317 leader_election.cc:288] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9 [CANDIDATE]: Term 1 election: Requested vote from peers bf3f337ceb234e2fbf26e507c009a87f (127.24.69.130:42795), 6621104ea62e411da47ab612f9f26e89 (127.24.69.131:37289)
I20240501 20:01:45.968678 29310 tablet_bootstrap.cc:654] T 2587cb164a134d5f9112e11a7176e223 P 6621104ea62e411da47ab612f9f26e89: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:45.985605 29334 raft_consensus.cc:2726] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 20:01:45.985368 29316 leader_election.cc:288] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [CANDIDATE]: Term 1 election: Requested vote from peers a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987), bf3f337ceb234e2fbf26e507c009a87f (127.24.69.130:42795)
I20240501 20:01:45.986449 29192 raft_consensus.cc:2454] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1a42f9b0ed84cc08e0819f874e79db9 in term 1.
I20240501 20:01:45.988765 29311 tablet_bootstrap.cc:492] T 2587cb164a134d5f9112e11a7176e223 P a1a42f9b0ed84cc08e0819f874e79db9: No bootstrap required, opened a new log
I20240501 20:01:45.988626 29081 leader_election.cc:302] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1a42f9b0ed84cc08e0819f874e79db9, bf3f337ceb234e2fbf26e507c009a87f; no voters:
I20240501 20:01:45.989542 29311 ts_tablet_manager.cc:1367] T 2587cb164a134d5f9112e11a7176e223 P a1a42f9b0ed84cc08e0819f874e79db9: Time spent bootstrapping tablet: real 0.085s user 0.024s sys 0.015s
I20240501 20:01:45.989766 29317 raft_consensus.cc:2781] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:45.990322 29317 raft_consensus.cc:686] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9 [term 1 LEADER]: Becoming Leader. State: Replica: a1a42f9b0ed84cc08e0819f874e79db9, State: Running, Role: LEADER
I20240501 20:01:45.990514 29312 tablet_bootstrap.cc:492] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f: No bootstrap required, opened a new log
I20240501 20:01:45.990988 29312 ts_tablet_manager.cc:1367] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f: Time spent bootstrapping tablet: real 0.125s user 0.051s sys 0.057s
I20240501 20:01:45.991463 29317 consensus_queue.cc:237] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:45.992770 29311 raft_consensus.cc:348] T 2587cb164a134d5f9112e11a7176e223 P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
W20240501 20:01:45.993472 29080 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.69.131:37289, user_credentials={real_user=slave}} blocked reactor thread for 47669.6us
I20240501 20:01:45.993592 29311 raft_consensus.cc:374] T 2587cb164a134d5f9112e11a7176e223 P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:45.993631 29312 raft_consensus.cc:348] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:45.994220 29311 raft_consensus.cc:729] T 2587cb164a134d5f9112e11a7176e223 P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a42f9b0ed84cc08e0819f874e79db9, State: Initialized, Role: FOLLOWER
I20240501 20:01:45.994671 29312 raft_consensus.cc:374] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:45.995095 29312 raft_consensus.cc:729] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf3f337ceb234e2fbf26e507c009a87f, State: Initialized, Role: FOLLOWER
I20240501 20:01:45.995210 29266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b9198949354d2e9202677a25193fa9" candidate_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6621104ea62e411da47ab612f9f26e89"
I20240501 20:01:45.995339 29311 consensus_queue.cc:260] T 2587cb164a134d5f9112e11a7176e223 P a1a42f9b0ed84cc08e0819f874e79db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:45.996062 29266 raft_consensus.cc:2379] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a1a42f9b0ed84cc08e0819f874e79db9 in current term 1: Already voted for candidate 6621104ea62e411da47ab612f9f26e89 in this term.
I20240501 20:01:45.996030 29312 consensus_queue.cc:260] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:45.997053 29310 tablet_bootstrap.cc:492] T 2587cb164a134d5f9112e11a7176e223 P 6621104ea62e411da47ab612f9f26e89: No bootstrap required, opened a new log
I20240501 20:01:45.997529 29310 ts_tablet_manager.cc:1367] T 2587cb164a134d5f9112e11a7176e223 P 6621104ea62e411da47ab612f9f26e89: Time spent bootstrapping tablet: real 0.071s user 0.009s sys 0.001s
I20240501 20:01:45.997762 29311 ts_tablet_manager.cc:1398] T 2587cb164a134d5f9112e11a7176e223 P a1a42f9b0ed84cc08e0819f874e79db9: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240501 20:01:46.000468 29312 ts_tablet_manager.cc:1398] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20240501 20:01:46.000063 29310 raft_consensus.cc:348] T 2587cb164a134d5f9112e11a7176e223 P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:46.000978 29310 raft_consensus.cc:374] T 2587cb164a134d5f9112e11a7176e223 P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:46.001354 29310 raft_consensus.cc:729] T 2587cb164a134d5f9112e11a7176e223 P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6621104ea62e411da47ab612f9f26e89, State: Initialized, Role: FOLLOWER
I20240501 20:01:46.002193 29310 consensus_queue.cc:260] T 2587cb164a134d5f9112e11a7176e223 P 6621104ea62e411da47ab612f9f26e89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:46.004768 29310 ts_tablet_manager.cc:1398] T 2587cb164a134d5f9112e11a7176e223 P 6621104ea62e411da47ab612f9f26e89: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240501 20:01:46.006755 29013 catalog_manager.cc:5484] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129). New cstate: current_term: 1 leader_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:46.036597 29334 raft_consensus.cc:482] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:46.037014 29334 raft_consensus.cc:504] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } }
I20240501 20:01:46.038626 29334 leader_election.cc:288] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987), bf3f337ceb234e2fbf26e507c009a87f (127.24.69.130:42795)
I20240501 20:01:46.039551 29117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82aaf53585ea4d87aa1d65dc40658d3f" candidate_uuid: "6621104ea62e411da47ab612f9f26e89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" is_pre_election: true
I20240501 20:01:46.039731 29192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82aaf53585ea4d87aa1d65dc40658d3f" candidate_uuid: "6621104ea62e411da47ab612f9f26e89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf3f337ceb234e2fbf26e507c009a87f" is_pre_election: true
I20240501 20:01:46.040232 29117 raft_consensus.cc:2454] T 82aaf53585ea4d87aa1d65dc40658d3f P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6621104ea62e411da47ab612f9f26e89 in term 0.
I20240501 20:01:46.040377 29192 raft_consensus.cc:2454] T 82aaf53585ea4d87aa1d65dc40658d3f P bf3f337ceb234e2fbf26e507c009a87f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6621104ea62e411da47ab612f9f26e89 in term 0.
I20240501 20:01:46.041414 29232 leader_election.cc:302] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6621104ea62e411da47ab612f9f26e89, a1a42f9b0ed84cc08e0819f874e79db9; no voters:
I20240501 20:01:46.042052 29334 raft_consensus.cc:2781] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:46.042305 29334 raft_consensus.cc:482] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:46.042507 29334 raft_consensus.cc:3037] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:46.047668 29334 raft_consensus.cc:504] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } }
I20240501 20:01:46.049126 29334 leader_election.cc:288] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [CANDIDATE]: Term 1 election: Requested vote from peers a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987), bf3f337ceb234e2fbf26e507c009a87f (127.24.69.130:42795)
I20240501 20:01:46.049882 29117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82aaf53585ea4d87aa1d65dc40658d3f" candidate_uuid: "6621104ea62e411da47ab612f9f26e89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a42f9b0ed84cc08e0819f874e79db9"
I20240501 20:01:46.049971 29192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82aaf53585ea4d87aa1d65dc40658d3f" candidate_uuid: "6621104ea62e411da47ab612f9f26e89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf3f337ceb234e2fbf26e507c009a87f"
I20240501 20:01:46.050351 29117 raft_consensus.cc:3037] T 82aaf53585ea4d87aa1d65dc40658d3f P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:46.050508 29192 raft_consensus.cc:3037] T 82aaf53585ea4d87aa1d65dc40658d3f P bf3f337ceb234e2fbf26e507c009a87f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:46.054745 29192 raft_consensus.cc:2454] T 82aaf53585ea4d87aa1d65dc40658d3f P bf3f337ceb234e2fbf26e507c009a87f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6621104ea62e411da47ab612f9f26e89 in term 1.
I20240501 20:01:46.055405 29117 raft_consensus.cc:2454] T 82aaf53585ea4d87aa1d65dc40658d3f P a1a42f9b0ed84cc08e0819f874e79db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6621104ea62e411da47ab612f9f26e89 in term 1.
I20240501 20:01:46.055601 29230 leader_election.cc:302] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6621104ea62e411da47ab612f9f26e89, bf3f337ceb234e2fbf26e507c009a87f; no voters:
I20240501 20:01:46.056424 29334 raft_consensus.cc:2781] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:46.056866 29334 raft_consensus.cc:686] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [term 1 LEADER]: Becoming Leader. State: Replica: 6621104ea62e411da47ab612f9f26e89, State: Running, Role: LEADER
I20240501 20:01:46.057577 29334 consensus_queue.cc:237] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } }
I20240501 20:01:46.063851 29013 catalog_manager.cc:5484] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6621104ea62e411da47ab612f9f26e89 (127.24.69.131). New cstate: current_term: 1 leader_uuid: "6621104ea62e411da47ab612f9f26e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:46.154348 29318 raft_consensus.cc:482] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:46.154855 29318 raft_consensus.cc:504] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:46.156970 29318 leader_election.cc:288] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987), 6621104ea62e411da47ab612f9f26e89 (127.24.69.131:37289)
I20240501 20:01:46.166983 29117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2587cb164a134d5f9112e11a7176e223" candidate_uuid: "bf3f337ceb234e2fbf26e507c009a87f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" is_pre_election: true
I20240501 20:01:46.167200 29266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2587cb164a134d5f9112e11a7176e223" candidate_uuid: "bf3f337ceb234e2fbf26e507c009a87f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6621104ea62e411da47ab612f9f26e89" is_pre_election: true
I20240501 20:01:46.167569 29117 raft_consensus.cc:2454] T 2587cb164a134d5f9112e11a7176e223 P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf3f337ceb234e2fbf26e507c009a87f in term 0.
I20240501 20:01:46.167783 29266 raft_consensus.cc:2454] T 2587cb164a134d5f9112e11a7176e223 P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf3f337ceb234e2fbf26e507c009a87f in term 0.
I20240501 20:01:46.168614 29158 leader_election.cc:302] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1a42f9b0ed84cc08e0819f874e79db9, bf3f337ceb234e2fbf26e507c009a87f; no voters:
I20240501 20:01:46.169252 29318 raft_consensus.cc:2781] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:46.169514 29318 raft_consensus.cc:482] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:46.169772 29318 raft_consensus.cc:3037] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:46.174108 29318 raft_consensus.cc:504] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:46.175477 29318 leader_election.cc:288] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [CANDIDATE]: Term 1 election: Requested vote from peers a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987), 6621104ea62e411da47ab612f9f26e89 (127.24.69.131:37289)
I20240501 20:01:46.176167 29117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2587cb164a134d5f9112e11a7176e223" candidate_uuid: "bf3f337ceb234e2fbf26e507c009a87f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1a42f9b0ed84cc08e0819f874e79db9"
I20240501 20:01:46.176417 29266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2587cb164a134d5f9112e11a7176e223" candidate_uuid: "bf3f337ceb234e2fbf26e507c009a87f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6621104ea62e411da47ab612f9f26e89"
I20240501 20:01:46.176843 29117 raft_consensus.cc:3037] T 2587cb164a134d5f9112e11a7176e223 P a1a42f9b0ed84cc08e0819f874e79db9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:46.176960 29266 raft_consensus.cc:3037] T 2587cb164a134d5f9112e11a7176e223 P 6621104ea62e411da47ab612f9f26e89 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:46.183048 29266 raft_consensus.cc:2454] T 2587cb164a134d5f9112e11a7176e223 P 6621104ea62e411da47ab612f9f26e89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf3f337ceb234e2fbf26e507c009a87f in term 1.
I20240501 20:01:46.183048 29117 raft_consensus.cc:2454] T 2587cb164a134d5f9112e11a7176e223 P a1a42f9b0ed84cc08e0819f874e79db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf3f337ceb234e2fbf26e507c009a87f in term 1.
I20240501 20:01:46.184322 29155 leader_election.cc:302] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6621104ea62e411da47ab612f9f26e89, bf3f337ceb234e2fbf26e507c009a87f; no voters:
I20240501 20:01:46.184942 29318 raft_consensus.cc:2781] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:46.185776 29318 raft_consensus.cc:686] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [term 1 LEADER]: Becoming Leader. State: Replica: bf3f337ceb234e2fbf26e507c009a87f, State: Running, Role: LEADER
I20240501 20:01:46.186484 29318 consensus_queue.cc:237] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:46.192435 29013 catalog_manager.cc:5484] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f reported cstate change: term changed from 0 to 1, leader changed from <none> to bf3f337ceb234e2fbf26e507c009a87f (127.24.69.130). New cstate: current_term: 1 leader_uuid: "bf3f337ceb234e2fbf26e507c009a87f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:46.241623 24854 tablet_server.cc:178] TabletServer@127.24.69.129:0 shutting down...
I20240501 20:01:46.265810 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:01:46.267071 24854 tablet_replica.cc:330] T 2587cb164a134d5f9112e11a7176e223 P a1a42f9b0ed84cc08e0819f874e79db9: stopping tablet replica
I20240501 20:01:46.267840 24854 raft_consensus.cc:2229] T 2587cb164a134d5f9112e11a7176e223 P a1a42f9b0ed84cc08e0819f874e79db9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:46.268378 24854 raft_consensus.cc:2258] T 2587cb164a134d5f9112e11a7176e223 P a1a42f9b0ed84cc08e0819f874e79db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:46.270808 24854 tablet_replica.cc:330] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9: stopping tablet replica
I20240501 20:01:46.271497 24854 raft_consensus.cc:2229] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:01:46.272231 24854 pending_rounds.cc:62] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9: Trying to abort 1 pending ops.
I20240501 20:01:46.272492 24854 pending_rounds.cc:68] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022975819750903808 op_type: NO_OP noop_request { }
I20240501 20:01:46.272898 24854 raft_consensus.cc:2866] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240501 20:01:46.273226 24854 raft_consensus.cc:2258] T 27b9198949354d2e9202677a25193fa9 P a1a42f9b0ed84cc08e0819f874e79db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:46.275871 24854 tablet_replica.cc:330] T 82aaf53585ea4d87aa1d65dc40658d3f P a1a42f9b0ed84cc08e0819f874e79db9: stopping tablet replica
I20240501 20:01:46.276620 24854 raft_consensus.cc:2229] T 82aaf53585ea4d87aa1d65dc40658d3f P a1a42f9b0ed84cc08e0819f874e79db9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:46.277156 24854 raft_consensus.cc:2258] T 82aaf53585ea4d87aa1d65dc40658d3f P a1a42f9b0ed84cc08e0819f874e79db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:46.279251 24854 tablet_replica.cc:330] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9: stopping tablet replica
I20240501 20:01:46.279804 24854 raft_consensus.cc:2229] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:01:46.280548 24854 pending_rounds.cc:62] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9: Trying to abort 1 pending ops.
I20240501 20:01:46.280802 24854 pending_rounds.cc:68] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022975819378311168 op_type: NO_OP noop_request { }
I20240501 20:01:46.281155 24854 raft_consensus.cc:2866] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240501 20:01:46.281481 24854 raft_consensus.cc:2258] T 128e1005f1df4010982d9ebf0c02f727 P a1a42f9b0ed84cc08e0819f874e79db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:46.298542 24854 tablet_server.cc:194] TabletServer@127.24.69.129:0 shutdown complete.
I20240501 20:01:46.536166 29334 consensus_queue.cc:1035] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:01:46.547039 29232 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44987: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240501 20:01:46.551647 29232 consensus_peers.cc:485] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 -> Peer a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987): Couldn't send request to peer a1a42f9b0ed84cc08e0819f874e79db9. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:46.790439 29318 consensus_queue.cc:1035] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240501 20:01:46.811820 29158 consensus_peers.cc:485] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f -> Peer a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987): Couldn't send request to peer a1a42f9b0ed84cc08e0819f874e79db9. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:47.274646 29318 consensus_queue.cc:579] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [LEADER]: Leader has been unable to successfully communicate with peer a1a42f9b0ed84cc08e0819f874e79db9 for more than 1 seconds (1.087s)
W20240501 20:01:47.281172 29015 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2587cb164a134d5f9112e11a7176e223 with cas_config_opid_index -1: no extra replica candidate found for tablet 2587cb164a134d5f9112e11a7176e223 (table test-workload [id=bff1774076514b6b9b88f12033448b1e]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20240501 20:01:47.310810 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:47.318715 29354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:47.319804 29355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:47.320885 29357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:47.321559 24854 server_base.cc:1034] running on GCE node
I20240501 20:01:47.323529 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:01:47.323882 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:47.324072 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593707324057 us; error 0 us; skew 500 ppm
I20240501 20:01:47.324673 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:47.327443 24854 webserver.cc:460] Webserver started at http://127.24.69.132:45041/ using document root <none> and password file <none>
I20240501 20:01:47.327932 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:47.328130 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:47.328440 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:47.329824 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714593576104309-24854-0/minicluster-data/ts-3-root/instance:
uuid: "91c35ea5bdb2414e8cf2ae2c96949167"
format_stamp: "Formatted at 2024-05-01 20:01:47 on dist-test-slave-z8x3"
I20240501 20:01:47.334442 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 20:01:47.338552 29362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:47.339315 24854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:01:47.339576 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714593576104309-24854-0/minicluster-data/ts-3-root
uuid: "91c35ea5bdb2414e8cf2ae2c96949167"
format_stamp: "Formatted at 2024-05-01 20:01:47 on dist-test-slave-z8x3"
I20240501 20:01:47.339875 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714593576104309-24854-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714593576104309-24854-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1714593576104309-24854-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:47.355343 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:47.356639 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:47.358081 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:47.360488 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:47.360711 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:47.360934 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:47.361097 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:47.368391 29334 consensus_queue.cc:579] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Leader has been unable to successfully communicate with peer a1a42f9b0ed84cc08e0819f874e79db9 for more than 1 seconds (1.310s)
I20240501 20:01:47.409991 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.132:42561
I20240501 20:01:47.410101 29426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.132:42561 every 8 connection(s)
I20240501 20:01:47.425094 29427 heartbeater.cc:347] Connected to a master server at 127.24.69.190:44515
I20240501 20:01:47.425454 29427 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:47.426183 29427 heartbeater.cc:511] Master 127.24.69.190:44515 requested a full tablet report, sending...
I20240501 20:01:47.427924 29015 ts_manager.cc:194] Registered new tserver with Master: 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132:42561)
I20240501 20:01:47.429361 29015 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45516
I20240501 20:01:47.442276 29266 consensus_queue.cc:237] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: true } }
I20240501 20:01:47.448237 29192 raft_consensus.cc:1261] T 82aaf53585ea4d87aa1d65dc40658d3f P bf3f337ceb234e2fbf26e507c009a87f [term 1 FOLLOWER]: Refusing update from remote peer 6621104ea62e411da47ab612f9f26e89: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:47.450304 29371 consensus_queue.cc:1035] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:01:47.453729 29232 consensus_peers.cc:485] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 -> Peer a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987): Couldn't send request to peer a1a42f9b0ed84cc08e0819f874e79db9. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:47.456928 29334 raft_consensus.cc:2932] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: true } } }
I20240501 20:01:47.458618 29192 raft_consensus.cc:2932] T 82aaf53585ea4d87aa1d65dc40658d3f P bf3f337ceb234e2fbf26e507c009a87f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: true } } }
W20240501 20:01:47.461074 29232 consensus_peers.cc:485] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 -> Peer 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132:42561): Couldn't send request to peer 91c35ea5bdb2414e8cf2ae2c96949167. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82aaf53585ea4d87aa1d65dc40658d3f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:47.466212 28999 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 82aaf53585ea4d87aa1d65dc40658d3f with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 2)
I20240501 20:01:47.470032 29015 catalog_manager.cc:5484] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 reported cstate change: config changed from index -1 to 2, NON_VOTER 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) added. New cstate: current_term: 1 leader_uuid: "6621104ea62e411da47ab612f9f26e89" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:47.515383 29192 consensus_queue.cc:237] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: true } }
I20240501 20:01:47.520993 29266 raft_consensus.cc:1261] T 2587cb164a134d5f9112e11a7176e223 P 6621104ea62e411da47ab612f9f26e89 [term 1 FOLLOWER]: Refusing update from remote peer bf3f337ceb234e2fbf26e507c009a87f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:47.522557 29350 consensus_queue.cc:1035] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240501 20:01:47.525231 29158 consensus_peers.cc:485] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f -> Peer a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987): Couldn't send request to peer a1a42f9b0ed84cc08e0819f874e79db9. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:01:47.529582 29158 consensus_peers.cc:485] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f -> Peer 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132:42561): Couldn't send request to peer 91c35ea5bdb2414e8cf2ae2c96949167. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2587cb164a134d5f9112e11a7176e223. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:47.528800 29318 raft_consensus.cc:2932] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: true } } }
I20240501 20:01:47.530373 29266 raft_consensus.cc:2932] T 2587cb164a134d5f9112e11a7176e223 P 6621104ea62e411da47ab612f9f26e89 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: true } } }
I20240501 20:01:47.539258 29000 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2587cb164a134d5f9112e11a7176e223 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 4)
I20240501 20:01:47.544414 29013 catalog_manager.cc:5484] T 2587cb164a134d5f9112e11a7176e223 P 6621104ea62e411da47ab612f9f26e89 reported cstate change: config changed from index -1 to 2, NON_VOTER 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) added. New cstate: current_term: 1 leader_uuid: "bf3f337ceb234e2fbf26e507c009a87f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: true } } }
W20240501 20:01:47.565026 29063 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver a1a42f9b0ed84cc08e0819f874e79db9 not available for placement
I20240501 20:01:47.642602 29334 raft_consensus.cc:482] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:47.643137 29334 raft_consensus.cc:504] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:47.644678 29334 leader_election.cc:288] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987), bf3f337ceb234e2fbf26e507c009a87f (127.24.69.130:42795)
I20240501 20:01:47.645790 29192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b9198949354d2e9202677a25193fa9" candidate_uuid: "6621104ea62e411da47ab612f9f26e89" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf3f337ceb234e2fbf26e507c009a87f" is_pre_election: true
I20240501 20:01:47.646661 29192 raft_consensus.cc:2454] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6621104ea62e411da47ab612f9f26e89 in term 1.
I20240501 20:01:47.647882 29230 leader_election.cc:302] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6621104ea62e411da47ab612f9f26e89, bf3f337ceb234e2fbf26e507c009a87f; no voters:
W20240501 20:01:47.648460 29232 leader_election.cc:334] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987): Network error: Client connection negotiation failed: client connection to 127.24.69.129:44987: connect: Connection refused (error 111)
I20240501 20:01:47.648645 29334 raft_consensus.cc:2781] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:47.649044 29334 raft_consensus.cc:482] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:47.649295 29334 raft_consensus.cc:3037] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:47.653801 29334 raft_consensus.cc:504] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:47.655179 29334 leader_election.cc:288] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [CANDIDATE]: Term 2 election: Requested vote from peers a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987), bf3f337ceb234e2fbf26e507c009a87f (127.24.69.130:42795)
I20240501 20:01:47.656293 29192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b9198949354d2e9202677a25193fa9" candidate_uuid: "6621104ea62e411da47ab612f9f26e89" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf3f337ceb234e2fbf26e507c009a87f"
I20240501 20:01:47.656868 29192 raft_consensus.cc:3037] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:01:47.658408 29232 leader_election.cc:334] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987): Network error: Client connection negotiation failed: client connection to 127.24.69.129:44987: connect: Connection refused (error 111)
I20240501 20:01:47.661584 29192 raft_consensus.cc:2454] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6621104ea62e411da47ab612f9f26e89 in term 2.
I20240501 20:01:47.662416 29230 leader_election.cc:302] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6621104ea62e411da47ab612f9f26e89, bf3f337ceb234e2fbf26e507c009a87f; no voters: a1a42f9b0ed84cc08e0819f874e79db9
I20240501 20:01:47.663033 29334 raft_consensus.cc:2781] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:47.663429 29334 raft_consensus.cc:686] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [term 2 LEADER]: Becoming Leader. State: Replica: 6621104ea62e411da47ab612f9f26e89, State: Running, Role: LEADER
I20240501 20:01:47.664237 29334 consensus_queue.cc:237] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:47.671111 29015 catalog_manager.cc:5484] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 reported cstate change: term changed from 1 to 2, leader changed from a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129) to 6621104ea62e411da47ab612f9f26e89 (127.24.69.131). New cstate: current_term: 2 leader_uuid: "6621104ea62e411da47ab612f9f26e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:47.758893 29334 raft_consensus.cc:482] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:47.759425 29334 raft_consensus.cc:504] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:47.761022 29334 leader_election.cc:288] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987), bf3f337ceb234e2fbf26e507c009a87f (127.24.69.130:42795)
I20240501 20:01:47.762054 29192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128e1005f1df4010982d9ebf0c02f727" candidate_uuid: "6621104ea62e411da47ab612f9f26e89" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf3f337ceb234e2fbf26e507c009a87f" is_pre_election: true
I20240501 20:01:47.762719 29192 raft_consensus.cc:2454] T 128e1005f1df4010982d9ebf0c02f727 P bf3f337ceb234e2fbf26e507c009a87f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6621104ea62e411da47ab612f9f26e89 in term 1.
I20240501 20:01:47.763707 29230 leader_election.cc:302] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6621104ea62e411da47ab612f9f26e89, bf3f337ceb234e2fbf26e507c009a87f; no voters:
I20240501 20:01:47.764339 29334 raft_consensus.cc:2781] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240501 20:01:47.764425 29232 leader_election.cc:334] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987): Network error: Client connection negotiation failed: client connection to 127.24.69.129:44987: connect: Connection refused (error 111)
I20240501 20:01:47.764777 29334 raft_consensus.cc:482] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:47.765131 29334 raft_consensus.cc:3037] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:47.769877 29334 raft_consensus.cc:504] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:47.771327 29334 leader_election.cc:288] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [CANDIDATE]: Term 2 election: Requested vote from peers a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987), bf3f337ceb234e2fbf26e507c009a87f (127.24.69.130:42795)
I20240501 20:01:47.772322 29192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128e1005f1df4010982d9ebf0c02f727" candidate_uuid: "6621104ea62e411da47ab612f9f26e89" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf3f337ceb234e2fbf26e507c009a87f"
I20240501 20:01:47.772797 29192 raft_consensus.cc:3037] T 128e1005f1df4010982d9ebf0c02f727 P bf3f337ceb234e2fbf26e507c009a87f [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:01:47.774264 29232 leader_election.cc:334] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987): Network error: Client connection negotiation failed: client connection to 127.24.69.129:44987: connect: Connection refused (error 111)
I20240501 20:01:47.777163 29192 raft_consensus.cc:2454] T 128e1005f1df4010982d9ebf0c02f727 P bf3f337ceb234e2fbf26e507c009a87f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6621104ea62e411da47ab612f9f26e89 in term 2.
I20240501 20:01:47.778028 29230 leader_election.cc:302] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6621104ea62e411da47ab612f9f26e89, bf3f337ceb234e2fbf26e507c009a87f; no voters: a1a42f9b0ed84cc08e0819f874e79db9
I20240501 20:01:47.778636 29334 raft_consensus.cc:2781] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:47.778998 29334 raft_consensus.cc:686] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [term 2 LEADER]: Becoming Leader. State: Replica: 6621104ea62e411da47ab612f9f26e89, State: Running, Role: LEADER
I20240501 20:01:47.779742 29334 consensus_queue.cc:237] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } }
I20240501 20:01:47.785997 29015 catalog_manager.cc:5484] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 reported cstate change: term changed from 1 to 2, leader changed from a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129) to 6621104ea62e411da47ab612f9f26e89 (127.24.69.131). New cstate: current_term: 2 leader_uuid: "6621104ea62e411da47ab612f9f26e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:47.830106 29440 ts_tablet_manager.cc:902] T 82aaf53585ea4d87aa1d65dc40658d3f P 91c35ea5bdb2414e8cf2ae2c96949167: Initiating tablet copy from peer 6621104ea62e411da47ab612f9f26e89 (127.24.69.131:37289)
I20240501 20:01:47.833542 29440 tablet_copy_client.cc:308] T 82aaf53585ea4d87aa1d65dc40658d3f P 91c35ea5bdb2414e8cf2ae2c96949167: tablet copy: Beginning tablet copy session from remote peer at address 127.24.69.131:37289
I20240501 20:01:47.843333 29276 tablet_copy_service.cc:140] P 6621104ea62e411da47ab612f9f26e89: Received BeginTabletCopySession request for tablet 82aaf53585ea4d87aa1d65dc40658d3f from peer 91c35ea5bdb2414e8cf2ae2c96949167 ({username='slave'} at 127.0.0.1:58694)
I20240501 20:01:47.843839 29276 tablet_copy_service.cc:161] P 6621104ea62e411da47ab612f9f26e89: Beginning new tablet copy session on tablet 82aaf53585ea4d87aa1d65dc40658d3f from peer 91c35ea5bdb2414e8cf2ae2c96949167 at {username='slave'} at 127.0.0.1:58694: session id = 91c35ea5bdb2414e8cf2ae2c96949167-82aaf53585ea4d87aa1d65dc40658d3f
I20240501 20:01:47.849892 29276 tablet_copy_source_session.cc:205] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:01:47.852492 29440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82aaf53585ea4d87aa1d65dc40658d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:47.863500 29440 tablet_copy_client.cc:783] T 82aaf53585ea4d87aa1d65dc40658d3f P 91c35ea5bdb2414e8cf2ae2c96949167: tablet copy: Starting download of 0 data blocks...
I20240501 20:01:47.864065 29440 tablet_copy_client.cc:647] T 82aaf53585ea4d87aa1d65dc40658d3f P 91c35ea5bdb2414e8cf2ae2c96949167: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:47.867056 29440 tablet_copy_client.cc:523] T 82aaf53585ea4d87aa1d65dc40658d3f P 91c35ea5bdb2414e8cf2ae2c96949167: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:47.872696 29440 tablet_bootstrap.cc:492] T 82aaf53585ea4d87aa1d65dc40658d3f P 91c35ea5bdb2414e8cf2ae2c96949167: Bootstrap starting.
I20240501 20:01:47.886623 29440 tablet_bootstrap.cc:492] T 82aaf53585ea4d87aa1d65dc40658d3f P 91c35ea5bdb2414e8cf2ae2c96949167: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:01:47.887307 29440 tablet_bootstrap.cc:492] T 82aaf53585ea4d87aa1d65dc40658d3f P 91c35ea5bdb2414e8cf2ae2c96949167: Bootstrap complete.
I20240501 20:01:47.887853 29440 ts_tablet_manager.cc:1367] T 82aaf53585ea4d87aa1d65dc40658d3f P 91c35ea5bdb2414e8cf2ae2c96949167: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.006s
I20240501 20:01:47.890151 29440 raft_consensus.cc:348] T 82aaf53585ea4d87aa1d65dc40658d3f P 91c35ea5bdb2414e8cf2ae2c96949167 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: true } }
I20240501 20:01:47.890797 29440 raft_consensus.cc:729] T 82aaf53585ea4d87aa1d65dc40658d3f P 91c35ea5bdb2414e8cf2ae2c96949167 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91c35ea5bdb2414e8cf2ae2c96949167, State: Initialized, Role: LEARNER
I20240501 20:01:47.891320 29440 consensus_queue.cc:260] T 82aaf53585ea4d87aa1d65dc40658d3f P 91c35ea5bdb2414e8cf2ae2c96949167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: true } }
I20240501 20:01:47.893371 29427 heartbeater.cc:503] Master 127.24.69.190:44515 was elected leader, sending a full tablet report...
I20240501 20:01:47.893635 29440 ts_tablet_manager.cc:1398] T 82aaf53585ea4d87aa1d65dc40658d3f P 91c35ea5bdb2414e8cf2ae2c96949167: Time spent starting tablet: real 0.006s user 0.000s sys 0.008s
I20240501 20:01:47.895247 29276 tablet_copy_service.cc:342] P 6621104ea62e411da47ab612f9f26e89: Request end of tablet copy session 91c35ea5bdb2414e8cf2ae2c96949167-82aaf53585ea4d87aa1d65dc40658d3f received from {username='slave'} at 127.0.0.1:58694
I20240501 20:01:47.895596 29276 tablet_copy_service.cc:433] P 6621104ea62e411da47ab612f9f26e89: ending tablet copy session 91c35ea5bdb2414e8cf2ae2c96949167-82aaf53585ea4d87aa1d65dc40658d3f on tablet 82aaf53585ea4d87aa1d65dc40658d3f with peer 91c35ea5bdb2414e8cf2ae2c96949167
I20240501 20:01:47.927829 29440 ts_tablet_manager.cc:902] T 2587cb164a134d5f9112e11a7176e223 P 91c35ea5bdb2414e8cf2ae2c96949167: Initiating tablet copy from peer bf3f337ceb234e2fbf26e507c009a87f (127.24.69.130:42795)
I20240501 20:01:47.930974 29440 tablet_copy_client.cc:308] T 2587cb164a134d5f9112e11a7176e223 P 91c35ea5bdb2414e8cf2ae2c96949167: tablet copy: Beginning tablet copy session from remote peer at address 127.24.69.130:42795
I20240501 20:01:47.939782 29202 tablet_copy_service.cc:140] P bf3f337ceb234e2fbf26e507c009a87f: Received BeginTabletCopySession request for tablet 2587cb164a134d5f9112e11a7176e223 from peer 91c35ea5bdb2414e8cf2ae2c96949167 ({username='slave'} at 127.0.0.1:45162)
I20240501 20:01:47.940296 29202 tablet_copy_service.cc:161] P bf3f337ceb234e2fbf26e507c009a87f: Beginning new tablet copy session on tablet 2587cb164a134d5f9112e11a7176e223 from peer 91c35ea5bdb2414e8cf2ae2c96949167 at {username='slave'} at 127.0.0.1:45162: session id = 91c35ea5bdb2414e8cf2ae2c96949167-2587cb164a134d5f9112e11a7176e223
I20240501 20:01:47.945880 29202 tablet_copy_source_session.cc:205] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:01:47.948491 29440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2587cb164a134d5f9112e11a7176e223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:47.959089 29440 tablet_copy_client.cc:783] T 2587cb164a134d5f9112e11a7176e223 P 91c35ea5bdb2414e8cf2ae2c96949167: tablet copy: Starting download of 0 data blocks...
I20240501 20:01:47.959582 29440 tablet_copy_client.cc:647] T 2587cb164a134d5f9112e11a7176e223 P 91c35ea5bdb2414e8cf2ae2c96949167: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:47.962733 29440 tablet_copy_client.cc:523] T 2587cb164a134d5f9112e11a7176e223 P 91c35ea5bdb2414e8cf2ae2c96949167: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:47.969278 29440 tablet_bootstrap.cc:492] T 2587cb164a134d5f9112e11a7176e223 P 91c35ea5bdb2414e8cf2ae2c96949167: Bootstrap starting.
I20240501 20:01:47.982614 29440 tablet_bootstrap.cc:492] T 2587cb164a134d5f9112e11a7176e223 P 91c35ea5bdb2414e8cf2ae2c96949167: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:01:47.983291 29440 tablet_bootstrap.cc:492] T 2587cb164a134d5f9112e11a7176e223 P 91c35ea5bdb2414e8cf2ae2c96949167: Bootstrap complete.
I20240501 20:01:47.983850 29440 ts_tablet_manager.cc:1367] T 2587cb164a134d5f9112e11a7176e223 P 91c35ea5bdb2414e8cf2ae2c96949167: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.002s
I20240501 20:01:47.985805 29440 raft_consensus.cc:348] T 2587cb164a134d5f9112e11a7176e223 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: true } }
I20240501 20:01:47.986395 29440 raft_consensus.cc:729] T 2587cb164a134d5f9112e11a7176e223 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91c35ea5bdb2414e8cf2ae2c96949167, State: Initialized, Role: LEARNER
I20240501 20:01:47.986971 29440 consensus_queue.cc:260] T 2587cb164a134d5f9112e11a7176e223 P 91c35ea5bdb2414e8cf2ae2c96949167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: true } }
I20240501 20:01:47.988555 29440 ts_tablet_manager.cc:1398] T 2587cb164a134d5f9112e11a7176e223 P 91c35ea5bdb2414e8cf2ae2c96949167: Time spent starting tablet: real 0.004s user 0.006s sys 0.002s
I20240501 20:01:47.989953 29202 tablet_copy_service.cc:342] P bf3f337ceb234e2fbf26e507c009a87f: Request end of tablet copy session 91c35ea5bdb2414e8cf2ae2c96949167-2587cb164a134d5f9112e11a7176e223 received from {username='slave'} at 127.0.0.1:45162
I20240501 20:01:47.990293 29202 tablet_copy_service.cc:433] P bf3f337ceb234e2fbf26e507c009a87f: ending tablet copy session 91c35ea5bdb2414e8cf2ae2c96949167-2587cb164a134d5f9112e11a7176e223 on tablet 2587cb164a134d5f9112e11a7176e223 with peer 91c35ea5bdb2414e8cf2ae2c96949167
W20240501 20:01:48.048859 29232 consensus_peers.cc:485] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 -> Peer a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987): Couldn't send request to peer a1a42f9b0ed84cc08e0819f874e79db9. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:48.080423 29192 raft_consensus.cc:1261] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f [term 2 FOLLOWER]: Refusing update from remote peer 6621104ea62e411da47ab612f9f26e89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240501 20:01:48.081686 29334 consensus_queue.cc:1035] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:48.247879 29192 raft_consensus.cc:1261] T 128e1005f1df4010982d9ebf0c02f727 P bf3f337ceb234e2fbf26e507c009a87f [term 2 FOLLOWER]: Refusing update from remote peer 6621104ea62e411da47ab612f9f26e89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240501 20:01:48.249080 29334 consensus_queue.cc:1035] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:01:48.262405 29232 consensus_peers.cc:485] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 -> Peer a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987): Couldn't send request to peer a1a42f9b0ed84cc08e0819f874e79db9. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:48.332374 29402 raft_consensus.cc:1203] T 2587cb164a134d5f9112e11a7176e223 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:01:48.442411 29402 raft_consensus.cc:1203] T 82aaf53585ea4d87aa1d65dc40658d3f P 91c35ea5bdb2414e8cf2ae2c96949167 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:01:48.530006 29458 consensus_queue.cc:579] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Leader has been unable to successfully communicate with peer a1a42f9b0ed84cc08e0819f874e79db9 for more than 1 seconds (1.087s)
W20240501 20:01:48.567049 29063 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver a1a42f9b0ed84cc08e0819f874e79db9 not available for placement
I20240501 20:01:48.827157 29458 consensus_queue.cc:579] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Leader has been unable to successfully communicate with peer a1a42f9b0ed84cc08e0819f874e79db9 for more than 1 seconds (1.046s)
I20240501 20:01:48.834897 29266 consensus_queue.cc:237] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: true } }
I20240501 20:01:48.839752 29192 raft_consensus.cc:1261] T 128e1005f1df4010982d9ebf0c02f727 P bf3f337ceb234e2fbf26e507c009a87f [term 2 FOLLOWER]: Refusing update from remote peer 6621104ea62e411da47ab612f9f26e89: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240501 20:01:48.840660 29232 consensus_peers.cc:485] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 -> Peer 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132:42561): Couldn't send request to peer 91c35ea5bdb2414e8cf2ae2c96949167. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 128e1005f1df4010982d9ebf0c02f727. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:48.841015 29458 consensus_queue.cc:1035] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:01:48.842778 29232 consensus_peers.cc:485] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 -> Peer a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987): Couldn't send request to peer a1a42f9b0ed84cc08e0819f874e79db9. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:48.846058 29373 raft_consensus.cc:2932] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: true } } }
I20240501 20:01:48.847301 29192 raft_consensus.cc:2932] T 128e1005f1df4010982d9ebf0c02f727 P bf3f337ceb234e2fbf26e507c009a87f [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: true } } }
I20240501 20:01:48.853109 28999 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 128e1005f1df4010982d9ebf0c02f727 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:01:48.857712 29015 catalog_manager.cc:5484] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 reported cstate change: config changed from index -1 to 2, NON_VOTER 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) added. New cstate: current_term: 2 leader_uuid: "6621104ea62e411da47ab612f9f26e89" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:48.860986 29463 raft_consensus.cc:1050] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89: attempting to promote NON_VOTER 91c35ea5bdb2414e8cf2ae2c96949167 to VOTER
I20240501 20:01:48.862707 29463 consensus_queue.cc:237] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } }
I20240501 20:01:48.869336 29402 raft_consensus.cc:1261] T 82aaf53585ea4d87aa1d65dc40658d3f P 91c35ea5bdb2414e8cf2ae2c96949167 [term 1 LEARNER]: Refusing update from remote peer 6621104ea62e411da47ab612f9f26e89: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:48.869501 29192 raft_consensus.cc:1261] T 82aaf53585ea4d87aa1d65dc40658d3f P bf3f337ceb234e2fbf26e507c009a87f [term 1 FOLLOWER]: Refusing update from remote peer 6621104ea62e411da47ab612f9f26e89: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:48.870990 29373 consensus_queue.cc:1035] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:48.871960 29458 consensus_queue.cc:1035] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 20:01:48.881016 29232 consensus_peers.cc:485] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 -> Peer a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987): Couldn't send request to peer a1a42f9b0ed84cc08e0819f874e79db9. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:48.886864 29458 raft_consensus.cc:2932] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } } }
I20240501 20:01:48.888428 29192 raft_consensus.cc:2932] T 82aaf53585ea4d87aa1d65dc40658d3f P bf3f337ceb234e2fbf26e507c009a87f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } } }
I20240501 20:01:48.896015 29401 raft_consensus.cc:2932] T 82aaf53585ea4d87aa1d65dc40658d3f P 91c35ea5bdb2414e8cf2ae2c96949167 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } } }
I20240501 20:01:48.898365 29013 catalog_manager.cc:5484] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 reported cstate change: config changed from index 2 to 3, 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6621104ea62e411da47ab612f9f26e89" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:48.905741 29460 raft_consensus.cc:1050] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f: attempting to promote NON_VOTER 91c35ea5bdb2414e8cf2ae2c96949167 to VOTER
I20240501 20:01:48.907968 29460 consensus_queue.cc:237] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } }
I20240501 20:01:48.912377 29266 consensus_queue.cc:237] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } }
I20240501 20:01:48.917399 29401 raft_consensus.cc:1261] T 2587cb164a134d5f9112e11a7176e223 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 1 LEARNER]: Refusing update from remote peer bf3f337ceb234e2fbf26e507c009a87f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:48.919621 29265 raft_consensus.cc:1261] T 2587cb164a134d5f9112e11a7176e223 P 6621104ea62e411da47ab612f9f26e89 [term 1 FOLLOWER]: Refusing update from remote peer bf3f337ceb234e2fbf26e507c009a87f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:48.919414 29473 consensus_queue.cc:1035] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:01:48.921316 29192 raft_consensus.cc:1261] T 82aaf53585ea4d87aa1d65dc40658d3f P bf3f337ceb234e2fbf26e507c009a87f [term 1 FOLLOWER]: Refusing update from remote peer 6621104ea62e411da47ab612f9f26e89: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:01:48.921725 29401 raft_consensus.cc:1261] T 82aaf53585ea4d87aa1d65dc40658d3f P 91c35ea5bdb2414e8cf2ae2c96949167 [term 1 FOLLOWER]: Refusing update from remote peer 6621104ea62e411da47ab612f9f26e89: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:01:48.921918 29471 consensus_queue.cc:1035] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:01:48.922964 29334 consensus_queue.cc:1035] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:48.924489 29334 consensus_queue.cc:1035] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:48.931355 29458 raft_consensus.cc:2932] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } } }
W20240501 20:01:48.932313 29158 consensus_peers.cc:485] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f -> Peer a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987): Couldn't send request to peer a1a42f9b0ed84cc08e0819f874e79db9. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:48.932736 29402 raft_consensus.cc:2932] T 82aaf53585ea4d87aa1d65dc40658d3f P 91c35ea5bdb2414e8cf2ae2c96949167 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } } }
I20240501 20:01:48.939687 29192 raft_consensus.cc:2932] T 82aaf53585ea4d87aa1d65dc40658d3f P bf3f337ceb234e2fbf26e507c009a87f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } } }
I20240501 20:01:48.940508 29460 raft_consensus.cc:2932] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } } }
I20240501 20:01:48.943266 28999 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 82aaf53585ea4d87aa1d65dc40658d3f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:01:48.944553 29401 raft_consensus.cc:2932] T 2587cb164a134d5f9112e11a7176e223 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } } }
I20240501 20:01:48.946964 29013 catalog_manager.cc:5484] T 82aaf53585ea4d87aa1d65dc40658d3f P 91c35ea5bdb2414e8cf2ae2c96949167 reported cstate change: config changed from index 3 to 4, VOTER a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129) evicted. New cstate: current_term: 1 leader_uuid: "6621104ea62e411da47ab612f9f26e89" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } } }
I20240501 20:01:48.957782 29265 raft_consensus.cc:2932] T 2587cb164a134d5f9112e11a7176e223 P 6621104ea62e411da47ab612f9f26e89 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } } }
I20240501 20:01:48.962924 29014 catalog_manager.cc:5484] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f reported cstate change: config changed from index 2 to 3, 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bf3f337ceb234e2fbf26e507c009a87f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:48.978555 29192 consensus_queue.cc:237] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } }
W20240501 20:01:48.979018 29002 catalog_manager.cc:4573] TS a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 82aaf53585ea4d87aa1d65dc40658d3f: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44987: connect: Connection refused (error 111)
I20240501 20:01:48.982710 29401 raft_consensus.cc:1261] T 2587cb164a134d5f9112e11a7176e223 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 1 FOLLOWER]: Refusing update from remote peer bf3f337ceb234e2fbf26e507c009a87f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:01:48.982867 29265 raft_consensus.cc:1261] T 2587cb164a134d5f9112e11a7176e223 P 6621104ea62e411da47ab612f9f26e89 [term 1 FOLLOWER]: Refusing update from remote peer bf3f337ceb234e2fbf26e507c009a87f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:01:48.983880 29460 consensus_queue.cc:1035] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:48.984447 29471 consensus_queue.cc:1035] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:48.990585 29474 raft_consensus.cc:2932] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } } }
I20240501 20:01:48.992035 29401 raft_consensus.cc:2932] T 2587cb164a134d5f9112e11a7176e223 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } } }
I20240501 20:01:48.992417 29265 raft_consensus.cc:2932] T 2587cb164a134d5f9112e11a7176e223 P 6621104ea62e411da47ab612f9f26e89 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } } }
I20240501 20:01:48.998134 29000 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2587cb164a134d5f9112e11a7176e223 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:01:49.002149 29013 catalog_manager.cc:5484] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f reported cstate change: config changed from index 3 to 4, VOTER a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129) evicted. New cstate: current_term: 1 leader_uuid: "bf3f337ceb234e2fbf26e507c009a87f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:49.051836 29334 consensus_queue.cc:579] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Leader has been unable to successfully communicate with peer a1a42f9b0ed84cc08e0819f874e79db9 for more than 1 seconds (1.387s)
I20240501 20:01:49.060323 29265 consensus_queue.cc:237] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: true } }
W20240501 20:01:49.065287 29232 consensus_peers.cc:485] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 -> Peer 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132:42561): Couldn't send request to peer 91c35ea5bdb2414e8cf2ae2c96949167. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27b9198949354d2e9202677a25193fa9. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:49.065443 29192 raft_consensus.cc:1261] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f [term 2 FOLLOWER]: Refusing update from remote peer 6621104ea62e411da47ab612f9f26e89: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:49.066679 29458 consensus_queue.cc:1035] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:01:49.067775 29232 consensus_peers.cc:485] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 -> Peer a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987): Couldn't send request to peer a1a42f9b0ed84cc08e0819f874e79db9. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:49.071646 29463 raft_consensus.cc:2932] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: true } } }
I20240501 20:01:49.073244 29192 raft_consensus.cc:2932] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: true } } }
I20240501 20:01:49.081077 28999 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 27b9198949354d2e9202677a25193fa9 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:01:49.085469 29014 catalog_manager.cc:5484] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f reported cstate change: config changed from index -1 to 2, NON_VOTER 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) added. New cstate: current_term: 2 leader_uuid: "6621104ea62e411da47ab612f9f26e89" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: true } } }
I20240501 20:01:49.328418 29479 ts_tablet_manager.cc:902] T 128e1005f1df4010982d9ebf0c02f727 P 91c35ea5bdb2414e8cf2ae2c96949167: Initiating tablet copy from peer 6621104ea62e411da47ab612f9f26e89 (127.24.69.131:37289)
I20240501 20:01:49.331570 29479 tablet_copy_client.cc:308] T 128e1005f1df4010982d9ebf0c02f727 P 91c35ea5bdb2414e8cf2ae2c96949167: tablet copy: Beginning tablet copy session from remote peer at address 127.24.69.131:37289
I20240501 20:01:49.332952 29276 tablet_copy_service.cc:140] P 6621104ea62e411da47ab612f9f26e89: Received BeginTabletCopySession request for tablet 128e1005f1df4010982d9ebf0c02f727 from peer 91c35ea5bdb2414e8cf2ae2c96949167 ({username='slave'} at 127.0.0.1:58694)
I20240501 20:01:49.333369 29276 tablet_copy_service.cc:161] P 6621104ea62e411da47ab612f9f26e89: Beginning new tablet copy session on tablet 128e1005f1df4010982d9ebf0c02f727 from peer 91c35ea5bdb2414e8cf2ae2c96949167 at {username='slave'} at 127.0.0.1:58694: session id = 91c35ea5bdb2414e8cf2ae2c96949167-128e1005f1df4010982d9ebf0c02f727
I20240501 20:01:49.338407 29276 tablet_copy_source_session.cc:205] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:01:49.340754 29479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128e1005f1df4010982d9ebf0c02f727. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:49.351428 29479 tablet_copy_client.cc:783] T 128e1005f1df4010982d9ebf0c02f727 P 91c35ea5bdb2414e8cf2ae2c96949167: tablet copy: Starting download of 0 data blocks...
I20240501 20:01:49.351823 29479 tablet_copy_client.cc:647] T 128e1005f1df4010982d9ebf0c02f727 P 91c35ea5bdb2414e8cf2ae2c96949167: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:49.354599 29479 tablet_copy_client.cc:523] T 128e1005f1df4010982d9ebf0c02f727 P 91c35ea5bdb2414e8cf2ae2c96949167: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:49.359829 29479 tablet_bootstrap.cc:492] T 128e1005f1df4010982d9ebf0c02f727 P 91c35ea5bdb2414e8cf2ae2c96949167: Bootstrap starting.
I20240501 20:01:49.373062 29479 tablet_bootstrap.cc:492] T 128e1005f1df4010982d9ebf0c02f727 P 91c35ea5bdb2414e8cf2ae2c96949167: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:01:49.373800 29479 tablet_bootstrap.cc:492] T 128e1005f1df4010982d9ebf0c02f727 P 91c35ea5bdb2414e8cf2ae2c96949167: Bootstrap complete.
I20240501 20:01:49.374266 29479 ts_tablet_manager.cc:1367] T 128e1005f1df4010982d9ebf0c02f727 P 91c35ea5bdb2414e8cf2ae2c96949167: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.004s
I20240501 20:01:49.375982 29479 raft_consensus.cc:348] T 128e1005f1df4010982d9ebf0c02f727 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: true } }
I20240501 20:01:49.376566 29479 raft_consensus.cc:729] T 128e1005f1df4010982d9ebf0c02f727 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 91c35ea5bdb2414e8cf2ae2c96949167, State: Initialized, Role: LEARNER
I20240501 20:01:49.377015 29479 consensus_queue.cc:260] T 128e1005f1df4010982d9ebf0c02f727 P 91c35ea5bdb2414e8cf2ae2c96949167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: true } }
I20240501 20:01:49.378494 29479 ts_tablet_manager.cc:1398] T 128e1005f1df4010982d9ebf0c02f727 P 91c35ea5bdb2414e8cf2ae2c96949167: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 20:01:49.379837 29276 tablet_copy_service.cc:342] P 6621104ea62e411da47ab612f9f26e89: Request end of tablet copy session 91c35ea5bdb2414e8cf2ae2c96949167-128e1005f1df4010982d9ebf0c02f727 received from {username='slave'} at 127.0.0.1:58694
I20240501 20:01:49.380138 29276 tablet_copy_service.cc:433] P 6621104ea62e411da47ab612f9f26e89: ending tablet copy session 91c35ea5bdb2414e8cf2ae2c96949167-128e1005f1df4010982d9ebf0c02f727 on tablet 128e1005f1df4010982d9ebf0c02f727 with peer 91c35ea5bdb2414e8cf2ae2c96949167
I20240501 20:01:49.517267 29479 ts_tablet_manager.cc:902] T 27b9198949354d2e9202677a25193fa9 P 91c35ea5bdb2414e8cf2ae2c96949167: Initiating tablet copy from peer 6621104ea62e411da47ab612f9f26e89 (127.24.69.131:37289)
I20240501 20:01:49.520306 29479 tablet_copy_client.cc:308] T 27b9198949354d2e9202677a25193fa9 P 91c35ea5bdb2414e8cf2ae2c96949167: tablet copy: Beginning tablet copy session from remote peer at address 127.24.69.131:37289
I20240501 20:01:49.521548 29276 tablet_copy_service.cc:140] P 6621104ea62e411da47ab612f9f26e89: Received BeginTabletCopySession request for tablet 27b9198949354d2e9202677a25193fa9 from peer 91c35ea5bdb2414e8cf2ae2c96949167 ({username='slave'} at 127.0.0.1:58694)
I20240501 20:01:49.521956 29276 tablet_copy_service.cc:161] P 6621104ea62e411da47ab612f9f26e89: Beginning new tablet copy session on tablet 27b9198949354d2e9202677a25193fa9 from peer 91c35ea5bdb2414e8cf2ae2c96949167 at {username='slave'} at 127.0.0.1:58694: session id = 91c35ea5bdb2414e8cf2ae2c96949167-27b9198949354d2e9202677a25193fa9
I20240501 20:01:49.526795 29276 tablet_copy_source_session.cc:205] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:01:49.529003 29479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b9198949354d2e9202677a25193fa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:49.538192 29479 tablet_copy_client.cc:783] T 27b9198949354d2e9202677a25193fa9 P 91c35ea5bdb2414e8cf2ae2c96949167: tablet copy: Starting download of 0 data blocks...
I20240501 20:01:49.538662 29479 tablet_copy_client.cc:647] T 27b9198949354d2e9202677a25193fa9 P 91c35ea5bdb2414e8cf2ae2c96949167: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:49.541536 29479 tablet_copy_client.cc:523] T 27b9198949354d2e9202677a25193fa9 P 91c35ea5bdb2414e8cf2ae2c96949167: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:49.546914 29479 tablet_bootstrap.cc:492] T 27b9198949354d2e9202677a25193fa9 P 91c35ea5bdb2414e8cf2ae2c96949167: Bootstrap starting.
I20240501 20:01:49.559690 29479 tablet_bootstrap.cc:492] T 27b9198949354d2e9202677a25193fa9 P 91c35ea5bdb2414e8cf2ae2c96949167: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:01:49.560271 29479 tablet_bootstrap.cc:492] T 27b9198949354d2e9202677a25193fa9 P 91c35ea5bdb2414e8cf2ae2c96949167: Bootstrap complete.
I20240501 20:01:49.560701 29479 ts_tablet_manager.cc:1367] T 27b9198949354d2e9202677a25193fa9 P 91c35ea5bdb2414e8cf2ae2c96949167: Time spent bootstrapping tablet: real 0.014s user 0.016s sys 0.000s
I20240501 20:01:49.562403 29479 raft_consensus.cc:348] T 27b9198949354d2e9202677a25193fa9 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: true } }
I20240501 20:01:49.562937 29479 raft_consensus.cc:729] T 27b9198949354d2e9202677a25193fa9 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 91c35ea5bdb2414e8cf2ae2c96949167, State: Initialized, Role: LEARNER
I20240501 20:01:49.563356 29479 consensus_queue.cc:260] T 27b9198949354d2e9202677a25193fa9 P 91c35ea5bdb2414e8cf2ae2c96949167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: true } }
I20240501 20:01:49.564936 29479 ts_tablet_manager.cc:1398] T 27b9198949354d2e9202677a25193fa9 P 91c35ea5bdb2414e8cf2ae2c96949167: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 20:01:49.566311 29276 tablet_copy_service.cc:342] P 6621104ea62e411da47ab612f9f26e89: Request end of tablet copy session 91c35ea5bdb2414e8cf2ae2c96949167-27b9198949354d2e9202677a25193fa9 received from {username='slave'} at 127.0.0.1:58694
I20240501 20:01:49.566597 29276 tablet_copy_service.cc:433] P 6621104ea62e411da47ab612f9f26e89: ending tablet copy session 91c35ea5bdb2414e8cf2ae2c96949167-27b9198949354d2e9202677a25193fa9 on tablet 27b9198949354d2e9202677a25193fa9 with peer 91c35ea5bdb2414e8cf2ae2c96949167
W20240501 20:01:49.570369 29063 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver a1a42f9b0ed84cc08e0819f874e79db9 not available for placement
I20240501 20:01:49.735793 29401 raft_consensus.cc:1203] T 128e1005f1df4010982d9ebf0c02f727 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20240501 20:01:49.946812 29458 consensus_queue.cc:579] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Leader has been unable to successfully communicate with peer a1a42f9b0ed84cc08e0819f874e79db9 for more than 1 seconds (1.111s)
I20240501 20:01:50.004835 29401 raft_consensus.cc:1203] T 27b9198949354d2e9202677a25193fa9 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20240501 20:01:50.111721 29458 consensus_queue.cc:579] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Leader has been unable to successfully communicate with peer a1a42f9b0ed84cc08e0819f874e79db9 for more than 1 seconds (1.050s)
W20240501 20:01:50.204799 29002 catalog_manager.cc:4573] TS a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 82aaf53585ea4d87aa1d65dc40658d3f: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44987: connect: Connection refused (error 111)
I20240501 20:01:50.223423 29458 raft_consensus.cc:1050] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89: attempting to promote NON_VOTER 91c35ea5bdb2414e8cf2ae2c96949167 to VOTER
I20240501 20:01:50.225587 29458 consensus_queue.cc:237] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } }
I20240501 20:01:50.230719 29192 raft_consensus.cc:1261] T 128e1005f1df4010982d9ebf0c02f727 P bf3f337ceb234e2fbf26e507c009a87f [term 2 FOLLOWER]: Refusing update from remote peer 6621104ea62e411da47ab612f9f26e89: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 20:01:50.230949 29401 raft_consensus.cc:1261] T 128e1005f1df4010982d9ebf0c02f727 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 2 LEARNER]: Refusing update from remote peer 6621104ea62e411da47ab612f9f26e89: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 20:01:50.232003 29495 consensus_queue.cc:1035] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:50.232949 29458 consensus_queue.cc:1035] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20240501 20:01:50.234419 29232 consensus_peers.cc:485] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 -> Peer a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987): Couldn't send request to peer a1a42f9b0ed84cc08e0819f874e79db9. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:50.240758 29495 raft_consensus.cc:2932] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } } }
I20240501 20:01:50.242048 29401 raft_consensus.cc:2932] T 128e1005f1df4010982d9ebf0c02f727 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } } }
I20240501 20:01:50.248626 29192 raft_consensus.cc:2932] T 128e1005f1df4010982d9ebf0c02f727 P bf3f337ceb234e2fbf26e507c009a87f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } } }
I20240501 20:01:50.251103 29015 catalog_manager.cc:5484] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 reported cstate change: config changed from index 2 to 3, 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6621104ea62e411da47ab612f9f26e89" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:50.265033 29265 consensus_queue.cc:237] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } }
I20240501 20:01:50.269138 29401 raft_consensus.cc:1261] T 128e1005f1df4010982d9ebf0c02f727 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 2 FOLLOWER]: Refusing update from remote peer 6621104ea62e411da47ab612f9f26e89: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:01:50.269266 29192 raft_consensus.cc:1261] T 128e1005f1df4010982d9ebf0c02f727 P bf3f337ceb234e2fbf26e507c009a87f [term 2 FOLLOWER]: Refusing update from remote peer 6621104ea62e411da47ab612f9f26e89: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:01:50.270304 29495 consensus_queue.cc:1035] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:50.270824 29490 consensus_queue.cc:1035] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:50.275967 29495 raft_consensus.cc:2932] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } } }
I20240501 20:01:50.277329 29192 raft_consensus.cc:2932] T 128e1005f1df4010982d9ebf0c02f727 P bf3f337ceb234e2fbf26e507c009a87f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } } }
I20240501 20:01:50.278468 29401 raft_consensus.cc:2932] T 128e1005f1df4010982d9ebf0c02f727 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } } }
I20240501 20:01:50.282672 28999 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 128e1005f1df4010982d9ebf0c02f727 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:01:50.287865 29013 catalog_manager.cc:5484] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 reported cstate change: config changed from index 3 to 4, VOTER a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129) evicted. New cstate: current_term: 2 leader_uuid: "6621104ea62e411da47ab612f9f26e89" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:50.529151 29495 raft_consensus.cc:1050] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89: attempting to promote NON_VOTER 91c35ea5bdb2414e8cf2ae2c96949167 to VOTER
I20240501 20:01:50.530877 29495 consensus_queue.cc:237] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } }
I20240501 20:01:50.536201 29401 raft_consensus.cc:1261] T 27b9198949354d2e9202677a25193fa9 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 2 LEARNER]: Refusing update from remote peer 6621104ea62e411da47ab612f9f26e89: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 20:01:50.536454 29192 raft_consensus.cc:1261] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f [term 2 FOLLOWER]: Refusing update from remote peer 6621104ea62e411da47ab612f9f26e89: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 20:01:50.537535 29490 consensus_queue.cc:1035] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:50.538250 29495 consensus_queue.cc:1035] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 20:01:50.539609 29232 consensus_peers.cc:485] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 -> Peer a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987): Couldn't send request to peer a1a42f9b0ed84cc08e0819f874e79db9. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:50.546078 29495 raft_consensus.cc:2932] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } } }
I20240501 20:01:50.547480 29401 raft_consensus.cc:2932] T 27b9198949354d2e9202677a25193fa9 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } } }
I20240501 20:01:50.554567 29192 raft_consensus.cc:2932] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } } }
I20240501 20:01:50.556860 29015 catalog_manager.cc:5484] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 reported cstate change: config changed from index 2 to 3, 91c35ea5bdb2414e8cf2ae2c96949167 (127.24.69.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6621104ea62e411da47ab612f9f26e89" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a1a42f9b0ed84cc08e0819f874e79db9" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:50.567752 29265 consensus_queue.cc:237] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } }
I20240501 20:01:50.572054 29192 raft_consensus.cc:1261] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f [term 2 FOLLOWER]: Refusing update from remote peer 6621104ea62e411da47ab612f9f26e89: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:01:50.573046 29490 consensus_queue.cc:1035] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:01:50.573626 29063 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver a1a42f9b0ed84cc08e0819f874e79db9 not available for placement
I20240501 20:01:50.574822 29401 raft_consensus.cc:1261] T 27b9198949354d2e9202677a25193fa9 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 2 FOLLOWER]: Refusing update from remote peer 6621104ea62e411da47ab612f9f26e89: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:01:50.576047 29490 consensus_queue.cc:1035] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:50.578469 29495 raft_consensus.cc:2932] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } } }
I20240501 20:01:50.580008 29192 raft_consensus.cc:2932] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } } }
I20240501 20:01:50.586298 28999 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 27b9198949354d2e9202677a25193fa9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:01:50.584865 29401 raft_consensus.cc:2932] T 27b9198949354d2e9202677a25193fa9 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } } }
I20240501 20:01:50.589928 29013 catalog_manager.cc:5484] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 reported cstate change: config changed from index 3 to 4, VOTER a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129) evicted. New cstate: current_term: 2 leader_uuid: "6621104ea62e411da47ab612f9f26e89" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bf3f337ceb234e2fbf26e507c009a87f" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 42795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6621104ea62e411da47ab612f9f26e89" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91c35ea5bdb2414e8cf2ae2c96949167" member_type: VOTER last_known_addr { host: "127.24.69.132" port: 42561 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 20:01:51.250967 29002 catalog_manager.cc:4573] TS a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2587cb164a134d5f9112e11a7176e223: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44987: connect: Connection refused (error 111)
W20240501 20:01:51.811980 29002 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44987: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 20:01:52.504113 29002 catalog_manager.cc:4573] TS a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 128e1005f1df4010982d9ebf0c02f727: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44987: connect: Connection refused (error 111)
I20240501 20:01:54.445420 24854 tablet_server.cc:178] TabletServer@127.24.69.130:0 shutting down...
I20240501 20:01:54.469164 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:01:54.469980 24854 tablet_replica.cc:330] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f: stopping tablet replica
I20240501 20:01:54.470824 24854 raft_consensus.cc:2229] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:01:54.471765 24854 raft_consensus.cc:2258] T 2587cb164a134d5f9112e11a7176e223 P bf3f337ceb234e2fbf26e507c009a87f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:54.475175 24854 tablet_replica.cc:330] T 128e1005f1df4010982d9ebf0c02f727 P bf3f337ceb234e2fbf26e507c009a87f: stopping tablet replica
I20240501 20:01:54.475682 24854 raft_consensus.cc:2229] T 128e1005f1df4010982d9ebf0c02f727 P bf3f337ceb234e2fbf26e507c009a87f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:54.476171 24854 raft_consensus.cc:2258] T 128e1005f1df4010982d9ebf0c02f727 P bf3f337ceb234e2fbf26e507c009a87f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:54.477865 24854 tablet_replica.cc:330] T 82aaf53585ea4d87aa1d65dc40658d3f P bf3f337ceb234e2fbf26e507c009a87f: stopping tablet replica
I20240501 20:01:54.478308 24854 raft_consensus.cc:2229] T 82aaf53585ea4d87aa1d65dc40658d3f P bf3f337ceb234e2fbf26e507c009a87f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:54.478789 24854 raft_consensus.cc:2258] T 82aaf53585ea4d87aa1d65dc40658d3f P bf3f337ceb234e2fbf26e507c009a87f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:54.480422 24854 tablet_replica.cc:330] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f: stopping tablet replica
I20240501 20:01:54.480821 24854 raft_consensus.cc:2229] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:54.481305 24854 raft_consensus.cc:2258] T 27b9198949354d2e9202677a25193fa9 P bf3f337ceb234e2fbf26e507c009a87f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:54.503551 24854 tablet_server.cc:194] TabletServer@127.24.69.130:0 shutdown complete.
I20240501 20:01:54.516048 24854 tablet_server.cc:178] TabletServer@127.24.69.131:0 shutting down...
I20240501 20:01:54.536404 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:01:54.537082 24854 tablet_replica.cc:330] T 2587cb164a134d5f9112e11a7176e223 P 6621104ea62e411da47ab612f9f26e89: stopping tablet replica
I20240501 20:01:54.537688 24854 raft_consensus.cc:2229] T 2587cb164a134d5f9112e11a7176e223 P 6621104ea62e411da47ab612f9f26e89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:54.538208 24854 raft_consensus.cc:2258] T 2587cb164a134d5f9112e11a7176e223 P 6621104ea62e411da47ab612f9f26e89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:54.539937 24854 tablet_replica.cc:330] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89: stopping tablet replica
I20240501 20:01:54.540405 24854 raft_consensus.cc:2229] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [term 2 LEADER]: Raft consensus shutting down.
I20240501 20:01:54.541198 24854 raft_consensus.cc:2258] T 27b9198949354d2e9202677a25193fa9 P 6621104ea62e411da47ab612f9f26e89 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:54.543049 24854 tablet_replica.cc:330] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89: stopping tablet replica
I20240501 20:01:54.543507 24854 raft_consensus.cc:2229] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:01:54.544201 24854 raft_consensus.cc:2258] T 82aaf53585ea4d87aa1d65dc40658d3f P 6621104ea62e411da47ab612f9f26e89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:54.546031 24854 tablet_replica.cc:330] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89: stopping tablet replica
I20240501 20:01:54.546463 24854 raft_consensus.cc:2229] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [term 2 LEADER]: Raft consensus shutting down.
I20240501 20:01:54.547106 24854 raft_consensus.cc:2258] T 128e1005f1df4010982d9ebf0c02f727 P 6621104ea62e411da47ab612f9f26e89 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:54.569321 24854 tablet_server.cc:194] TabletServer@127.24.69.131:0 shutdown complete.
I20240501 20:01:54.581418 24854 tablet_server.cc:178] TabletServer@127.24.69.132:0 shutting down...
I20240501 20:01:54.605397 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:01:54.606160 24854 tablet_replica.cc:330] T 27b9198949354d2e9202677a25193fa9 P 91c35ea5bdb2414e8cf2ae2c96949167: stopping tablet replica
I20240501 20:01:54.606720 24854 raft_consensus.cc:2229] T 27b9198949354d2e9202677a25193fa9 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:54.607192 24854 raft_consensus.cc:2258] T 27b9198949354d2e9202677a25193fa9 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240501 20:01:54.607403 29002 catalog_manager.cc:4573] TS a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 128e1005f1df4010982d9ebf0c02f727: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44987: connect: Connection refused (error 111)
I20240501 20:01:54.608918 24854 tablet_replica.cc:330] T 128e1005f1df4010982d9ebf0c02f727 P 91c35ea5bdb2414e8cf2ae2c96949167: stopping tablet replica
I20240501 20:01:54.609440 24854 raft_consensus.cc:2229] T 128e1005f1df4010982d9ebf0c02f727 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:54.609876 24854 raft_consensus.cc:2258] T 128e1005f1df4010982d9ebf0c02f727 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:54.611433 24854 tablet_replica.cc:330] T 2587cb164a134d5f9112e11a7176e223 P 91c35ea5bdb2414e8cf2ae2c96949167: stopping tablet replica
I20240501 20:01:54.611883 24854 raft_consensus.cc:2229] T 2587cb164a134d5f9112e11a7176e223 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:54.612365 24854 raft_consensus.cc:2258] T 2587cb164a134d5f9112e11a7176e223 P 91c35ea5bdb2414e8cf2ae2c96949167 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:54.613862 24854 tablet_replica.cc:330] T 82aaf53585ea4d87aa1d65dc40658d3f P 91c35ea5bdb2414e8cf2ae2c96949167: stopping tablet replica
I20240501 20:01:54.614290 24854 raft_consensus.cc:2229] T 82aaf53585ea4d87aa1d65dc40658d3f P 91c35ea5bdb2414e8cf2ae2c96949167 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:54.614722 24854 raft_consensus.cc:2258] T 82aaf53585ea4d87aa1d65dc40658d3f P 91c35ea5bdb2414e8cf2ae2c96949167 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:54.635677 24854 tablet_server.cc:194] TabletServer@127.24.69.132:0 shutdown complete.
I20240501 20:01:54.646052 24854 master.cc:537] Master@127.24.69.190:44515 shutting down...
W20240501 20:01:57.441471 29002 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44987: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20240501 20:01:57.444965 29002 catalog_manager.cc:4573] TS a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2587cb164a134d5f9112e11a7176e223: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44987: connect: Connection refused (error 111)
W20240501 20:01:58.715595 29002 catalog_manager.cc:4573] TS a1a42f9b0ed84cc08e0819f874e79db9 (127.24.69.129:44987): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 128e1005f1df4010982d9ebf0c02f727: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44987: connect: Connection refused (error 111)
I20240501 20:01:59.146430 24854 raft_consensus.cc:2229] T 00000000000000000000000000000000 P abb684196a634ebd9065a952ce95407c [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:01:59.147442 24854 raft_consensus.cc:2258] T 00000000000000000000000000000000 P abb684196a634ebd9065a952ce95407c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:59.147970 24854 tablet_replica.cc:330] T 00000000000000000000000000000000 P abb684196a634ebd9065a952ce95407c: stopping tablet replica
I20240501 20:01:59.176908 24854 master.cc:552] Master@127.24.69.190:44515 shutdown complete.
[ OK ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering (14787 ms)
[ RUN ] AutoRebalancerTest.TestHandlingFailedTservers
I20240501 20:01:59.221618 24854 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.69.190:35105
I20240501 20:01:59.222630 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:59.228611 29515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:59.229141 29516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:59.229971 24854 server_base.cc:1034] running on GCE node
W20240501 20:01:59.230480 29518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:59.231251 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:01:59.231457 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:59.231628 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593719231614 us; error 0 us; skew 500 ppm
I20240501 20:01:59.232120 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:59.234447 24854 webserver.cc:460] Webserver started at http://127.24.69.190:43615/ using document root <none> and password file <none>
I20240501 20:01:59.234900 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:59.235065 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:59.235329 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:59.236630 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/master-0-root/instance:
uuid: "3c19b51a8fbb4d1388b0a38650928c4d"
format_stamp: "Formatted at 2024-05-01 20:01:59 on dist-test-slave-z8x3"
I20240501 20:01:59.240774 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240501 20:01:59.243652 29523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:59.244349 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 20:01:59.244590 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/master-0-root
uuid: "3c19b51a8fbb4d1388b0a38650928c4d"
format_stamp: "Formatted at 2024-05-01 20:01:59 on dist-test-slave-z8x3"
I20240501 20:01:59.244850 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:59.272531 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:59.273687 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:59.311050 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.190:35105
I20240501 20:01:59.311151 29574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.190:35105 every 8 connection(s)
I20240501 20:01:59.314869 29575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:59.325819 29575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c19b51a8fbb4d1388b0a38650928c4d: Bootstrap starting.
I20240501 20:01:59.330003 29575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c19b51a8fbb4d1388b0a38650928c4d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:59.333761 29575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c19b51a8fbb4d1388b0a38650928c4d: No bootstrap required, opened a new log
I20240501 20:01:59.335587 29575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c19b51a8fbb4d1388b0a38650928c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c19b51a8fbb4d1388b0a38650928c4d" member_type: VOTER }
I20240501 20:01:59.336091 29575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c19b51a8fbb4d1388b0a38650928c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:59.336310 29575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c19b51a8fbb4d1388b0a38650928c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c19b51a8fbb4d1388b0a38650928c4d, State: Initialized, Role: FOLLOWER
I20240501 20:01:59.336786 29575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c19b51a8fbb4d1388b0a38650928c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c19b51a8fbb4d1388b0a38650928c4d" member_type: VOTER }
I20240501 20:01:59.337201 29575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c19b51a8fbb4d1388b0a38650928c4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:59.337386 29575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c19b51a8fbb4d1388b0a38650928c4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:59.337604 29575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c19b51a8fbb4d1388b0a38650928c4d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:59.341650 29575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c19b51a8fbb4d1388b0a38650928c4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c19b51a8fbb4d1388b0a38650928c4d" member_type: VOTER }
I20240501 20:01:59.342085 29575 leader_election.cc:302] T 00000000000000000000000000000000 P 3c19b51a8fbb4d1388b0a38650928c4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c19b51a8fbb4d1388b0a38650928c4d; no voters:
I20240501 20:01:59.343046 29575 leader_election.cc:288] T 00000000000000000000000000000000 P 3c19b51a8fbb4d1388b0a38650928c4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:59.343406 29578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c19b51a8fbb4d1388b0a38650928c4d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:59.344681 29578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c19b51a8fbb4d1388b0a38650928c4d [term 1 LEADER]: Becoming Leader. State: Replica: 3c19b51a8fbb4d1388b0a38650928c4d, State: Running, Role: LEADER
I20240501 20:01:59.345280 29578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c19b51a8fbb4d1388b0a38650928c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c19b51a8fbb4d1388b0a38650928c4d" member_type: VOTER }
I20240501 20:01:59.345959 29575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c19b51a8fbb4d1388b0a38650928c4d [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:59.348346 29580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c19b51a8fbb4d1388b0a38650928c4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c19b51a8fbb4d1388b0a38650928c4d. Latest consensus state: current_term: 1 leader_uuid: "3c19b51a8fbb4d1388b0a38650928c4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c19b51a8fbb4d1388b0a38650928c4d" member_type: VOTER } }
I20240501 20:01:59.348310 29579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c19b51a8fbb4d1388b0a38650928c4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c19b51a8fbb4d1388b0a38650928c4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c19b51a8fbb4d1388b0a38650928c4d" member_type: VOTER } }
I20240501 20:01:59.349074 29580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c19b51a8fbb4d1388b0a38650928c4d [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:59.349195 29579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c19b51a8fbb4d1388b0a38650928c4d [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:59.354583 29583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:59.362010 29583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:59.362936 24854 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 20:01:59.370366 29583 catalog_manager.cc:1347] Generated new cluster ID: c31558fda55b416e95e4ff0c079a90f5
I20240501 20:01:59.370630 29583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:59.386459 29583 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:59.387671 29583 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:59.403101 29583 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3c19b51a8fbb4d1388b0a38650928c4d: Generated new TSK 0
I20240501 20:01:59.403671 29583 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:01:59.429077 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:59.435145 29596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:59.437053 29597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:59.437745 29599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:59.438439 24854 server_base.cc:1034] running on GCE node
I20240501 20:01:59.439141 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:01:59.439337 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:59.439489 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593719439470 us; error 0 us; skew 500 ppm
I20240501 20:01:59.439992 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:59.442169 24854 webserver.cc:460] Webserver started at http://127.24.69.129:37839/ using document root <none> and password file <none>
I20240501 20:01:59.442593 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:59.442771 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:59.443007 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:59.444178 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-0-root/instance:
uuid: "e93d3d097d994a1992554c82c6881b83"
format_stamp: "Formatted at 2024-05-01 20:01:59 on dist-test-slave-z8x3"
I20240501 20:01:59.449025 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240501 20:01:59.452001 29604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:59.452781 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:01:59.453114 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-0-root
uuid: "e93d3d097d994a1992554c82c6881b83"
format_stamp: "Formatted at 2024-05-01 20:01:59 on dist-test-slave-z8x3"
I20240501 20:01:59.453361 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:59.471477 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:59.472504 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:59.473856 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:59.475975 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:59.476159 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:59.476388 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:59.476548 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:59.516075 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.129:44161
I20240501 20:01:59.516173 29666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.129:44161 every 8 connection(s)
I20240501 20:01:59.520547 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:59.528617 29671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:59.529635 29672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:59.534364 29674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:59.535243 29667 heartbeater.cc:347] Connected to a master server at 127.24.69.190:35105
I20240501 20:01:59.535554 29667 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:59.536473 24854 server_base.cc:1034] running on GCE node
I20240501 20:01:59.536451 29667 heartbeater.cc:511] Master 127.24.69.190:35105 requested a full tablet report, sending...
I20240501 20:01:59.537601 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:01:59.537880 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:59.538038 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593719538022 us; error 0 us; skew 500 ppm
I20240501 20:01:59.538843 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:59.538831 29540 ts_manager.cc:194] Registered new tserver with Master: e93d3d097d994a1992554c82c6881b83 (127.24.69.129:44161)
I20240501 20:01:59.541499 29540 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45064
I20240501 20:01:59.541796 24854 webserver.cc:460] Webserver started at http://127.24.69.130:44845/ using document root <none> and password file <none>
I20240501 20:01:59.542415 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:59.542652 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:59.542970 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:59.544174 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-1-root/instance:
uuid: "1e7eecb723634a5da85a7c4d19108130"
format_stamp: "Formatted at 2024-05-01 20:01:59 on dist-test-slave-z8x3"
I20240501 20:01:59.548681 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240501 20:01:59.551702 29679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:59.552431 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 20:01:59.552690 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-1-root
uuid: "1e7eecb723634a5da85a7c4d19108130"
format_stamp: "Formatted at 2024-05-01 20:01:59 on dist-test-slave-z8x3"
I20240501 20:01:59.552955 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:59.584491 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:59.585629 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:59.587129 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:59.589246 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:59.589437 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:59.589653 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:59.589807 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:59.627432 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.130:40933
I20240501 20:01:59.627537 29741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.130:40933 every 8 connection(s)
I20240501 20:01:59.631958 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:59.639554 29745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:59.640902 29746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:59.642125 29748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:59.643285 24854 server_base.cc:1034] running on GCE node
I20240501 20:01:59.643487 29742 heartbeater.cc:347] Connected to a master server at 127.24.69.190:35105
I20240501 20:01:59.643900 29742 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:59.644170 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:01:59.644438 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:59.644593 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593719644576 us; error 0 us; skew 500 ppm
I20240501 20:01:59.644646 29742 heartbeater.cc:511] Master 127.24.69.190:35105 requested a full tablet report, sending...
I20240501 20:01:59.645231 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:59.646835 29540 ts_manager.cc:194] Registered new tserver with Master: 1e7eecb723634a5da85a7c4d19108130 (127.24.69.130:40933)
I20240501 20:01:59.648028 24854 webserver.cc:460] Webserver started at http://127.24.69.131:38977/ using document root <none> and password file <none>
I20240501 20:01:59.648350 29540 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45070
I20240501 20:01:59.648555 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:59.648801 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:59.649053 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:59.650336 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-2-root/instance:
uuid: "5947681784a745eea2d6b0c97e234a4c"
format_stamp: "Formatted at 2024-05-01 20:01:59 on dist-test-slave-z8x3"
I20240501 20:01:59.654711 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 20:01:59.657894 29753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:59.658564 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:01:59.658826 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-2-root
uuid: "5947681784a745eea2d6b0c97e234a4c"
format_stamp: "Formatted at 2024-05-01 20:01:59 on dist-test-slave-z8x3"
I20240501 20:01:59.659085 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:59.671404 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:59.672497 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:59.673818 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:59.675884 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:59.676074 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:59.676316 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:59.676471 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:59.713927 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.131:43209
I20240501 20:01:59.714021 29815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.131:43209 every 8 connection(s)
I20240501 20:01:59.726306 29816 heartbeater.cc:347] Connected to a master server at 127.24.69.190:35105
I20240501 20:01:59.726632 29816 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:59.727349 29816 heartbeater.cc:511] Master 127.24.69.190:35105 requested a full tablet report, sending...
I20240501 20:01:59.729038 29540 ts_manager.cc:194] Registered new tserver with Master: 5947681784a745eea2d6b0c97e234a4c (127.24.69.131:43209)
I20240501 20:01:59.729161 24854 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012229988s
I20240501 20:01:59.730331 29540 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45078
I20240501 20:02:00.544196 29667 heartbeater.cc:503] Master 127.24.69.190:35105 was elected leader, sending a full tablet report...
I20240501 20:02:00.650529 29742 heartbeater.cc:503] Master 127.24.69.190:35105 was elected leader, sending a full tablet report...
I20240501 20:02:00.732968 29816 heartbeater.cc:503] Master 127.24.69.190:35105 was elected leader, sending a full tablet report...
I20240501 20:02:00.760939 24854 test_util.cc:268] Using random seed: -173475253
I20240501 20:02:00.783159 29540 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45082:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 20:02:00.785487 29540 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:02:00.835884 29707 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3d39a2ed344577b6712279908a54ff (DEFAULT_TABLE table=test-workload [id=dc10af045d674afc9a7552af1c9866a9]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240501 20:02:00.837407 29707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3d39a2ed344577b6712279908a54ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:00.839906 29706 tablet_service.cc:1467] Processing CreateTablet for tablet ed6ad07388d449cf8a8504a00f60dbd8 (DEFAULT_TABLE table=test-workload [id=dc10af045d674afc9a7552af1c9866a9]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240501 20:02:00.841415 29706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed6ad07388d449cf8a8504a00f60dbd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:00.843904 29705 tablet_service.cc:1467] Processing CreateTablet for tablet 96a2762d729f41a3a34b013bf76d4702 (DEFAULT_TABLE table=test-workload [id=dc10af045d674afc9a7552af1c9866a9]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240501 20:02:00.845233 29705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96a2762d729f41a3a34b013bf76d4702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:00.845680 29630 tablet_service.cc:1467] Processing CreateTablet for tablet 96a2762d729f41a3a34b013bf76d4702 (DEFAULT_TABLE table=test-workload [id=dc10af045d674afc9a7552af1c9866a9]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240501 20:02:00.847013 29630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96a2762d729f41a3a34b013bf76d4702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:00.846647 29631 tablet_service.cc:1467] Processing CreateTablet for tablet ed6ad07388d449cf8a8504a00f60dbd8 (DEFAULT_TABLE table=test-workload [id=dc10af045d674afc9a7552af1c9866a9]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240501 20:02:00.848891 29631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed6ad07388d449cf8a8504a00f60dbd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:00.849058 29629 tablet_service.cc:1467] Processing CreateTablet for tablet cd88640130a342d899a02a89c3a9fcfa (DEFAULT_TABLE table=test-workload [id=dc10af045d674afc9a7552af1c9866a9]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240501 20:02:00.844882 29632 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3d39a2ed344577b6712279908a54ff (DEFAULT_TABLE table=test-workload [id=dc10af045d674afc9a7552af1c9866a9]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240501 20:02:00.850529 29629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd88640130a342d899a02a89c3a9fcfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:00.853209 29632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3d39a2ed344577b6712279908a54ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:00.849002 29704 tablet_service.cc:1467] Processing CreateTablet for tablet cd88640130a342d899a02a89c3a9fcfa (DEFAULT_TABLE table=test-workload [id=dc10af045d674afc9a7552af1c9866a9]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240501 20:02:00.856894 29704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd88640130a342d899a02a89c3a9fcfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:00.864639 29781 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3d39a2ed344577b6712279908a54ff (DEFAULT_TABLE table=test-workload [id=dc10af045d674afc9a7552af1c9866a9]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240501 20:02:00.865368 29778 tablet_service.cc:1467] Processing CreateTablet for tablet cd88640130a342d899a02a89c3a9fcfa (DEFAULT_TABLE table=test-workload [id=dc10af045d674afc9a7552af1c9866a9]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240501 20:02:00.865180 29780 tablet_service.cc:1467] Processing CreateTablet for tablet ed6ad07388d449cf8a8504a00f60dbd8 (DEFAULT_TABLE table=test-workload [id=dc10af045d674afc9a7552af1c9866a9]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240501 20:02:00.867193 29781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3d39a2ed344577b6712279908a54ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:00.867411 29779 tablet_service.cc:1467] Processing CreateTablet for tablet 96a2762d729f41a3a34b013bf76d4702 (DEFAULT_TABLE table=test-workload [id=dc10af045d674afc9a7552af1c9866a9]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240501 20:02:00.867889 29780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed6ad07388d449cf8a8504a00f60dbd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:00.868757 29779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96a2762d729f41a3a34b013bf76d4702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:00.878038 29778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd88640130a342d899a02a89c3a9fcfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:00.903666 29836 tablet_bootstrap.cc:492] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83: Bootstrap starting.
I20240501 20:02:00.909202 29836 tablet_bootstrap.cc:654] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:00.929440 29837 tablet_bootstrap.cc:492] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c: Bootstrap starting.
I20240501 20:02:00.932047 29838 tablet_bootstrap.cc:492] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130: Bootstrap starting.
I20240501 20:02:00.932974 29836 tablet_bootstrap.cc:492] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83: No bootstrap required, opened a new log
I20240501 20:02:00.938134 29836 ts_tablet_manager.cc:1367] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83: Time spent bootstrapping tablet: real 0.035s user 0.015s sys 0.014s
I20240501 20:02:00.939141 29838 tablet_bootstrap.cc:654] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:00.939909 29837 tablet_bootstrap.cc:654] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:00.940973 29836 raft_consensus.cc:348] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:00.941792 29836 raft_consensus.cc:374] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:00.942137 29836 raft_consensus.cc:729] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e93d3d097d994a1992554c82c6881b83, State: Initialized, Role: FOLLOWER
I20240501 20:02:00.943132 29836 consensus_queue.cc:260] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:00.944396 29838 tablet_bootstrap.cc:492] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130: No bootstrap required, opened a new log
I20240501 20:02:00.945001 29838 ts_tablet_manager.cc:1367] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20240501 20:02:00.945577 29837 tablet_bootstrap.cc:492] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c: No bootstrap required, opened a new log
I20240501 20:02:00.946035 29836 ts_tablet_manager.cc:1398] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20240501 20:02:00.946149 29837 ts_tablet_manager.cc:1367] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.000s
I20240501 20:02:00.947171 29836 tablet_bootstrap.cc:492] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83: Bootstrap starting.
I20240501 20:02:00.947891 29838 raft_consensus.cc:348] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:00.948730 29838 raft_consensus.cc:374] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:00.949081 29838 raft_consensus.cc:729] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e7eecb723634a5da85a7c4d19108130, State: Initialized, Role: FOLLOWER
I20240501 20:02:00.949035 29837 raft_consensus.cc:348] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:00.950201 29837 raft_consensus.cc:374] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:00.950167 29838 consensus_queue.cc:260] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:00.950758 29837 raft_consensus.cc:729] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5947681784a745eea2d6b0c97e234a4c, State: Initialized, Role: FOLLOWER
I20240501 20:02:00.951790 29837 consensus_queue.cc:260] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:00.953440 29836 tablet_bootstrap.cc:654] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:00.961928 29838 ts_tablet_manager.cc:1398] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130: Time spent starting tablet: real 0.017s user 0.008s sys 0.004s
I20240501 20:02:00.962865 29838 tablet_bootstrap.cc:492] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130: Bootstrap starting.
I20240501 20:02:00.970852 29837 ts_tablet_manager.cc:1398] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c: Time spent starting tablet: real 0.024s user 0.012s sys 0.008s
I20240501 20:02:00.972231 29837 tablet_bootstrap.cc:492] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c: Bootstrap starting.
I20240501 20:02:00.977721 29838 tablet_bootstrap.cc:654] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:00.979952 29837 tablet_bootstrap.cc:654] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:00.980671 29836 tablet_bootstrap.cc:492] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83: No bootstrap required, opened a new log
I20240501 20:02:00.981098 29836 ts_tablet_manager.cc:1367] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.019s
I20240501 20:02:00.983834 29838 tablet_bootstrap.cc:492] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130: No bootstrap required, opened a new log
I20240501 20:02:00.983764 29836 raft_consensus.cc:348] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:00.984455 29838 ts_tablet_manager.cc:1367] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.002s
I20240501 20:02:00.984591 29836 raft_consensus.cc:374] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:00.984961 29836 raft_consensus.cc:729] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e93d3d097d994a1992554c82c6881b83, State: Initialized, Role: FOLLOWER
I20240501 20:02:00.985402 29837 tablet_bootstrap.cc:492] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c: No bootstrap required, opened a new log
I20240501 20:02:00.985894 29837 ts_tablet_manager.cc:1367] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20240501 20:02:00.985678 29836 consensus_queue.cc:260] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:00.987231 29838 raft_consensus.cc:348] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:00.987867 29836 ts_tablet_manager.cc:1398] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 20:02:00.987987 29838 raft_consensus.cc:374] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:00.988426 29838 raft_consensus.cc:729] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e7eecb723634a5da85a7c4d19108130, State: Initialized, Role: FOLLOWER
I20240501 20:02:00.988922 29836 tablet_bootstrap.cc:492] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83: Bootstrap starting.
I20240501 20:02:00.988562 29837 raft_consensus.cc:348] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:00.989352 29837 raft_consensus.cc:374] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:00.989285 29838 consensus_queue.cc:260] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:00.989809 29837 raft_consensus.cc:729] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5947681784a745eea2d6b0c97e234a4c, State: Initialized, Role: FOLLOWER
I20240501 20:02:00.991896 29838 ts_tablet_manager.cc:1398] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240501 20:02:00.992853 29838 tablet_bootstrap.cc:492] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130: Bootstrap starting.
I20240501 20:02:00.994938 29836 tablet_bootstrap.cc:654] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:00.998535 29838 tablet_bootstrap.cc:654] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:01.002285 29837 consensus_queue.cc:260] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:01.004416 29838 tablet_bootstrap.cc:492] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130: No bootstrap required, opened a new log
I20240501 20:02:01.004542 29837 ts_tablet_manager.cc:1398] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20240501 20:02:01.004973 29838 ts_tablet_manager.cc:1367] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.004s
I20240501 20:02:01.005489 29837 tablet_bootstrap.cc:492] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c: Bootstrap starting.
I20240501 20:02:01.007726 29838 raft_consensus.cc:348] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:01.008598 29838 raft_consensus.cc:374] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:01.008980 29838 raft_consensus.cc:729] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e7eecb723634a5da85a7c4d19108130, State: Initialized, Role: FOLLOWER
I20240501 20:02:01.009850 29838 consensus_queue.cc:260] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:01.011597 29837 tablet_bootstrap.cc:654] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:01.014940 29836 tablet_bootstrap.cc:492] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83: No bootstrap required, opened a new log
I20240501 20:02:01.015463 29836 ts_tablet_manager.cc:1367] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.016s
I20240501 20:02:01.017539 29838 ts_tablet_manager.cc:1398] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130: Time spent starting tablet: real 0.012s user 0.002s sys 0.004s
I20240501 20:02:01.018069 29836 raft_consensus.cc:348] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:01.019135 29836 raft_consensus.cc:374] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:01.019496 29836 raft_consensus.cc:729] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e93d3d097d994a1992554c82c6881b83, State: Initialized, Role: FOLLOWER
I20240501 20:02:01.020373 29836 consensus_queue.cc:260] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:01.022156 29836 ts_tablet_manager.cc:1398] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 20:02:01.024677 29836 tablet_bootstrap.cc:492] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83: Bootstrap starting.
I20240501 20:02:01.026446 29838 tablet_bootstrap.cc:492] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130: Bootstrap starting.
I20240501 20:02:01.029206 29837 tablet_bootstrap.cc:492] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c: No bootstrap required, opened a new log
I20240501 20:02:01.029736 29837 ts_tablet_manager.cc:1367] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.016s
I20240501 20:02:01.031244 29836 tablet_bootstrap.cc:654] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:01.032616 29838 tablet_bootstrap.cc:654] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:01.032531 29837 raft_consensus.cc:348] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:01.033524 29837 raft_consensus.cc:374] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:01.033883 29837 raft_consensus.cc:729] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5947681784a745eea2d6b0c97e234a4c, State: Initialized, Role: FOLLOWER
I20240501 20:02:01.034736 29837 consensus_queue.cc:260] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:01.037079 29837 ts_tablet_manager.cc:1398] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240501 20:02:01.038058 29837 tablet_bootstrap.cc:492] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c: Bootstrap starting.
I20240501 20:02:01.043795 29837 tablet_bootstrap.cc:654] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:01.047406 29836 tablet_bootstrap.cc:492] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83: No bootstrap required, opened a new log
I20240501 20:02:01.047451 29838 tablet_bootstrap.cc:492] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130: No bootstrap required, opened a new log
I20240501 20:02:01.047987 29838 ts_tablet_manager.cc:1367] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.012s
I20240501 20:02:01.047987 29836 ts_tablet_manager.cc:1367] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.018s
I20240501 20:02:01.049921 29837 tablet_bootstrap.cc:492] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c: No bootstrap required, opened a new log
I20240501 20:02:01.050302 29837 ts_tablet_manager.cc:1367] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.009s
I20240501 20:02:01.050911 29838 raft_consensus.cc:348] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:01.051108 29836 raft_consensus.cc:348] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:01.051710 29838 raft_consensus.cc:374] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:01.051829 29836 raft_consensus.cc:374] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:01.052074 29838 raft_consensus.cc:729] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e7eecb723634a5da85a7c4d19108130, State: Initialized, Role: FOLLOWER
I20240501 20:02:01.052153 29836 raft_consensus.cc:729] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e93d3d097d994a1992554c82c6881b83, State: Initialized, Role: FOLLOWER
I20240501 20:02:01.052171 29837 raft_consensus.cc:348] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:01.053184 29837 raft_consensus.cc:374] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:01.052886 29838 consensus_queue.cc:260] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:01.053524 29837 raft_consensus.cc:729] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5947681784a745eea2d6b0c97e234a4c, State: Initialized, Role: FOLLOWER
I20240501 20:02:01.053313 29836 consensus_queue.cc:260] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:01.054304 29837 consensus_queue.cc:260] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:01.055419 29838 ts_tablet_manager.cc:1398] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 20:02:01.057507 29836 ts_tablet_manager.cc:1398] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83: Time spent starting tablet: real 0.009s user 0.004s sys 0.001s
I20240501 20:02:01.058939 29837 ts_tablet_manager.cc:1398] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20240501 20:02:01.116245 29842 raft_consensus.cc:482] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:01.116719 29842 raft_consensus.cc:504] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:01.118700 29842 leader_election.cc:288] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e7eecb723634a5da85a7c4d19108130 (127.24.69.130:40933), 5947681784a745eea2d6b0c97e234a4c (127.24.69.131:43209)
I20240501 20:02:01.125465 29844 raft_consensus.cc:482] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:01.126003 29844 raft_consensus.cc:504] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:01.129544 29717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed6ad07388d449cf8a8504a00f60dbd8" candidate_uuid: "e93d3d097d994a1992554c82c6881b83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e7eecb723634a5da85a7c4d19108130" is_pre_election: true
I20240501 20:02:01.130475 29717 raft_consensus.cc:2454] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e93d3d097d994a1992554c82c6881b83 in term 0.
I20240501 20:02:01.130999 29791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed6ad07388d449cf8a8504a00f60dbd8" candidate_uuid: "e93d3d097d994a1992554c82c6881b83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5947681784a745eea2d6b0c97e234a4c" is_pre_election: true
I20240501 20:02:01.129865 29844 leader_election.cc:288] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e93d3d097d994a1992554c82c6881b83 (127.24.69.129:44161), 1e7eecb723634a5da85a7c4d19108130 (127.24.69.130:40933)
I20240501 20:02:01.131711 29791 raft_consensus.cc:2454] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e93d3d097d994a1992554c82c6881b83 in term 0.
I20240501 20:02:01.132020 29606 leader_election.cc:302] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e7eecb723634a5da85a7c4d19108130, e93d3d097d994a1992554c82c6881b83; no voters:
I20240501 20:02:01.134790 29842 raft_consensus.cc:2781] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:02:01.135154 29842 raft_consensus.cc:482] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:02:01.135499 29842 raft_consensus.cc:3037] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:01.140286 29642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a2762d729f41a3a34b013bf76d4702" candidate_uuid: "5947681784a745eea2d6b0c97e234a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e93d3d097d994a1992554c82c6881b83" is_pre_election: true
I20240501 20:02:01.140985 29642 raft_consensus.cc:2454] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5947681784a745eea2d6b0c97e234a4c in term 0.
I20240501 20:02:01.142244 29755 leader_election.cc:302] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5947681784a745eea2d6b0c97e234a4c, e93d3d097d994a1992554c82c6881b83; no voters:
I20240501 20:02:01.142005 29842 raft_consensus.cc:504] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:01.142944 29844 raft_consensus.cc:2781] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:02:01.143322 29844 raft_consensus.cc:482] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:02:01.143723 29844 raft_consensus.cc:3037] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:01.143865 29717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a2762d729f41a3a34b013bf76d4702" candidate_uuid: "5947681784a745eea2d6b0c97e234a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e7eecb723634a5da85a7c4d19108130" is_pre_election: true
I20240501 20:02:01.144462 29717 raft_consensus.cc:2454] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5947681784a745eea2d6b0c97e234a4c in term 0.
I20240501 20:02:01.145126 29842 leader_election.cc:288] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [CANDIDATE]: Term 1 election: Requested vote from peers 1e7eecb723634a5da85a7c4d19108130 (127.24.69.130:40933), 5947681784a745eea2d6b0c97e234a4c (127.24.69.131:43209)
I20240501 20:02:01.145568 29716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed6ad07388d449cf8a8504a00f60dbd8" candidate_uuid: "e93d3d097d994a1992554c82c6881b83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e7eecb723634a5da85a7c4d19108130"
I20240501 20:02:01.146199 29716 raft_consensus.cc:3037] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:01.146240 29791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed6ad07388d449cf8a8504a00f60dbd8" candidate_uuid: "e93d3d097d994a1992554c82c6881b83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5947681784a745eea2d6b0c97e234a4c"
I20240501 20:02:01.146869 29791 raft_consensus.cc:3037] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:01.149611 29844 raft_consensus.cc:504] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:01.151103 29844 leader_election.cc:288] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [CANDIDATE]: Term 1 election: Requested vote from peers e93d3d097d994a1992554c82c6881b83 (127.24.69.129:44161), 1e7eecb723634a5da85a7c4d19108130 (127.24.69.130:40933)
I20240501 20:02:01.151826 29642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a2762d729f41a3a34b013bf76d4702" candidate_uuid: "5947681784a745eea2d6b0c97e234a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e93d3d097d994a1992554c82c6881b83"
I20240501 20:02:01.152410 29642 raft_consensus.cc:3037] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:01.152614 29717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a2762d729f41a3a34b013bf76d4702" candidate_uuid: "5947681784a745eea2d6b0c97e234a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e7eecb723634a5da85a7c4d19108130"
I20240501 20:02:01.153162 29717 raft_consensus.cc:3037] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:01.153452 29716 raft_consensus.cc:2454] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e93d3d097d994a1992554c82c6881b83 in term 1.
I20240501 20:02:01.154028 29791 raft_consensus.cc:2454] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e93d3d097d994a1992554c82c6881b83 in term 1.
I20240501 20:02:01.154729 29606 leader_election.cc:302] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e7eecb723634a5da85a7c4d19108130, e93d3d097d994a1992554c82c6881b83; no voters:
I20240501 20:02:01.156281 29842 raft_consensus.cc:2781] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:02:01.159374 29642 raft_consensus.cc:2454] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5947681784a745eea2d6b0c97e234a4c in term 1.
I20240501 20:02:01.159819 29717 raft_consensus.cc:2454] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5947681784a745eea2d6b0c97e234a4c in term 1.
I20240501 20:02:01.159926 29842 raft_consensus.cc:686] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 1 LEADER]: Becoming Leader. State: Replica: e93d3d097d994a1992554c82c6881b83, State: Running, Role: LEADER
I20240501 20:02:01.160640 29755 leader_election.cc:302] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5947681784a745eea2d6b0c97e234a4c, e93d3d097d994a1992554c82c6881b83; no voters:
I20240501 20:02:01.161487 29844 raft_consensus.cc:2781] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:02:01.161520 29842 consensus_queue.cc:237] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:01.162918 29844 raft_consensus.cc:686] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [term 1 LEADER]: Becoming Leader. State: Replica: 5947681784a745eea2d6b0c97e234a4c, State: Running, Role: LEADER
I20240501 20:02:01.163719 29844 consensus_queue.cc:237] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:01.168776 29539 catalog_manager.cc:5484] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 reported cstate change: term changed from 0 to 1, leader changed from <none> to e93d3d097d994a1992554c82c6881b83 (127.24.69.129). New cstate: current_term: 1 leader_uuid: "e93d3d097d994a1992554c82c6881b83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } health_report { overall_health: UNKNOWN } } }
I20240501 20:02:01.171890 29842 raft_consensus.cc:482] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:01.171926 29537 catalog_manager.cc:5484] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5947681784a745eea2d6b0c97e234a4c (127.24.69.131). New cstate: current_term: 1 leader_uuid: "5947681784a745eea2d6b0c97e234a4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } health_report { overall_health: HEALTHY } } }
I20240501 20:02:01.172566 29842 raft_consensus.cc:504] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:01.174393 29842 leader_election.cc:288] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e7eecb723634a5da85a7c4d19108130 (127.24.69.130:40933), 5947681784a745eea2d6b0c97e234a4c (127.24.69.131:43209)
I20240501 20:02:01.175012 29717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3d39a2ed344577b6712279908a54ff" candidate_uuid: "e93d3d097d994a1992554c82c6881b83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e7eecb723634a5da85a7c4d19108130" is_pre_election: true
I20240501 20:02:01.175714 29717 raft_consensus.cc:2454] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e93d3d097d994a1992554c82c6881b83 in term 0.
I20240501 20:02:01.175769 29791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3d39a2ed344577b6712279908a54ff" candidate_uuid: "e93d3d097d994a1992554c82c6881b83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5947681784a745eea2d6b0c97e234a4c" is_pre_election: true
I20240501 20:02:01.176466 29791 raft_consensus.cc:2454] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e93d3d097d994a1992554c82c6881b83 in term 0.
I20240501 20:02:01.177600 29608 leader_election.cc:302] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5947681784a745eea2d6b0c97e234a4c, e93d3d097d994a1992554c82c6881b83; no voters:
I20240501 20:02:01.178337 29842 raft_consensus.cc:2781] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:02:01.178593 29842 raft_consensus.cc:482] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:02:01.178923 29842 raft_consensus.cc:3037] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:01.184103 29842 raft_consensus.cc:504] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:01.185482 29842 leader_election.cc:288] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [CANDIDATE]: Term 1 election: Requested vote from peers 1e7eecb723634a5da85a7c4d19108130 (127.24.69.130:40933), 5947681784a745eea2d6b0c97e234a4c (127.24.69.131:43209)
I20240501 20:02:01.186240 29717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3d39a2ed344577b6712279908a54ff" candidate_uuid: "e93d3d097d994a1992554c82c6881b83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e7eecb723634a5da85a7c4d19108130"
I20240501 20:02:01.186347 29791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3d39a2ed344577b6712279908a54ff" candidate_uuid: "e93d3d097d994a1992554c82c6881b83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5947681784a745eea2d6b0c97e234a4c"
I20240501 20:02:01.186728 29717 raft_consensus.cc:3037] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:01.186822 29791 raft_consensus.cc:3037] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:01.190995 29717 raft_consensus.cc:2454] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e93d3d097d994a1992554c82c6881b83 in term 1.
I20240501 20:02:01.191112 29791 raft_consensus.cc:2454] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e93d3d097d994a1992554c82c6881b83 in term 1.
I20240501 20:02:01.191926 29606 leader_election.cc:302] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e7eecb723634a5da85a7c4d19108130, e93d3d097d994a1992554c82c6881b83; no voters:
I20240501 20:02:01.192574 29842 raft_consensus.cc:2781] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:02:01.192934 29842 raft_consensus.cc:686] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [term 1 LEADER]: Becoming Leader. State: Replica: e93d3d097d994a1992554c82c6881b83, State: Running, Role: LEADER
I20240501 20:02:01.193619 29842 consensus_queue.cc:237] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:01.199524 29537 catalog_manager.cc:5484] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 reported cstate change: term changed from 0 to 1, leader changed from <none> to e93d3d097d994a1992554c82c6881b83 (127.24.69.129). New cstate: current_term: 1 leader_uuid: "e93d3d097d994a1992554c82c6881b83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } health_report { overall_health: UNKNOWN } } }
W20240501 20:02:01.360067 29588 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 20:02:01.363358 29844 raft_consensus.cc:482] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:01.363751 29844 raft_consensus.cc:504] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:01.365182 29844 leader_election.cc:288] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e93d3d097d994a1992554c82c6881b83 (127.24.69.129:44161), 1e7eecb723634a5da85a7c4d19108130 (127.24.69.130:40933)
I20240501 20:02:01.366029 29642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd88640130a342d899a02a89c3a9fcfa" candidate_uuid: "5947681784a745eea2d6b0c97e234a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e93d3d097d994a1992554c82c6881b83" is_pre_election: true
I20240501 20:02:01.366139 29717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd88640130a342d899a02a89c3a9fcfa" candidate_uuid: "5947681784a745eea2d6b0c97e234a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e7eecb723634a5da85a7c4d19108130" is_pre_election: true
I20240501 20:02:01.366581 29642 raft_consensus.cc:2454] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5947681784a745eea2d6b0c97e234a4c in term 0.
I20240501 20:02:01.366689 29717 raft_consensus.cc:2454] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5947681784a745eea2d6b0c97e234a4c in term 0.
I20240501 20:02:01.367512 29755 leader_election.cc:302] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5947681784a745eea2d6b0c97e234a4c, e93d3d097d994a1992554c82c6881b83; no voters:
I20240501 20:02:01.368180 29844 raft_consensus.cc:2781] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:02:01.368525 29844 raft_consensus.cc:482] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:02:01.368772 29844 raft_consensus.cc:3037] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:01.373231 29844 raft_consensus.cc:504] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:01.374600 29844 leader_election.cc:288] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [CANDIDATE]: Term 1 election: Requested vote from peers e93d3d097d994a1992554c82c6881b83 (127.24.69.129:44161), 1e7eecb723634a5da85a7c4d19108130 (127.24.69.130:40933)
I20240501 20:02:01.375420 29642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd88640130a342d899a02a89c3a9fcfa" candidate_uuid: "5947681784a745eea2d6b0c97e234a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e93d3d097d994a1992554c82c6881b83"
I20240501 20:02:01.375617 29717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd88640130a342d899a02a89c3a9fcfa" candidate_uuid: "5947681784a745eea2d6b0c97e234a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e7eecb723634a5da85a7c4d19108130"
I20240501 20:02:01.376029 29642 raft_consensus.cc:3037] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:01.376142 29717 raft_consensus.cc:3037] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:01.382356 29642 raft_consensus.cc:2454] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5947681784a745eea2d6b0c97e234a4c in term 1.
I20240501 20:02:01.382356 29717 raft_consensus.cc:2454] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5947681784a745eea2d6b0c97e234a4c in term 1.
I20240501 20:02:01.383486 29755 leader_election.cc:302] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5947681784a745eea2d6b0c97e234a4c, e93d3d097d994a1992554c82c6881b83; no voters:
I20240501 20:02:01.384114 29844 raft_consensus.cc:2781] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:02:01.384608 29844 raft_consensus.cc:686] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 1 LEADER]: Becoming Leader. State: Replica: 5947681784a745eea2d6b0c97e234a4c, State: Running, Role: LEADER
I20240501 20:02:01.385315 29844 consensus_queue.cc:237] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:01.391201 29537 catalog_manager.cc:5484] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5947681784a745eea2d6b0c97e234a4c (127.24.69.131). New cstate: current_term: 1 leader_uuid: "5947681784a745eea2d6b0c97e234a4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } health_report { overall_health: HEALTHY } } }
I20240501 20:02:01.444361 24854 tablet_server.cc:178] TabletServer@127.24.69.129:0 shutting down...
I20240501 20:02:01.460980 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:02:01.461717 24854 tablet_replica.cc:330] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83: stopping tablet replica
I20240501 20:02:01.462291 24854 raft_consensus.cc:2229] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:01.462666 24854 raft_consensus.cc:2258] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:01.464428 24854 tablet_replica.cc:330] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83: stopping tablet replica
I20240501 20:02:01.464900 24854 raft_consensus.cc:2229] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:02:01.465404 24854 pending_rounds.cc:62] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83: Trying to abort 1 pending ops.
I20240501 20:02:01.465591 24854 pending_rounds.cc:68] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022975881885708288 op_type: NO_OP noop_request { }
I20240501 20:02:01.465884 24854 raft_consensus.cc:2866] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240501 20:02:01.466123 24854 raft_consensus.cc:2258] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:01.467975 24854 tablet_replica.cc:330] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83: stopping tablet replica
I20240501 20:02:01.468493 24854 raft_consensus.cc:2229] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:02:01.468991 24854 pending_rounds.cc:62] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83: Trying to abort 1 pending ops.
I20240501 20:02:01.469158 24854 pending_rounds.cc:68] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022975882014990336 op_type: NO_OP noop_request { }
I20240501 20:02:01.469408 24854 raft_consensus.cc:2866] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240501 20:02:01.469637 24854 raft_consensus.cc:2258] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:01.471596 24854 tablet_replica.cc:330] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83: stopping tablet replica
I20240501 20:02:01.472023 24854 raft_consensus.cc:2229] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:01.472369 24854 raft_consensus.cc:2258] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:01.492846 24854 tablet_server.cc:194] TabletServer@127.24.69.129:0 shutdown complete.
I20240501 20:02:01.503371 24854 tablet_server.cc:178] TabletServer@127.24.69.130:0 shutting down...
I20240501 20:02:01.518337 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:02:01.519086 24854 tablet_replica.cc:330] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130: stopping tablet replica
I20240501 20:02:01.519711 24854 raft_consensus.cc:2229] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:01.520084 24854 raft_consensus.cc:2258] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:01.522037 24854 tablet_replica.cc:330] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130: stopping tablet replica
I20240501 20:02:01.522527 24854 raft_consensus.cc:2229] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:01.522965 24854 raft_consensus.cc:2258] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:01.524623 24854 tablet_replica.cc:330] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130: stopping tablet replica
I20240501 20:02:01.525105 24854 raft_consensus.cc:2229] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:01.525430 24854 raft_consensus.cc:2258] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:01.526857 24854 tablet_replica.cc:330] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130: stopping tablet replica
I20240501 20:02:01.527271 24854 raft_consensus.cc:2229] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:01.527585 24854 raft_consensus.cc:2258] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:01.548125 24854 tablet_server.cc:194] TabletServer@127.24.69.130:0 shutdown complete.
I20240501 20:02:01.559530 24854 tablet_server.cc:178] TabletServer@127.24.69.131:0 shutting down...
W20240501 20:02:01.562700 29755 consensus_peers.cc:485] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c -> Peer 1e7eecb723634a5da85a7c4d19108130 (127.24.69.130:40933): Couldn't send request to peer 1e7eecb723634a5da85a7c4d19108130. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.130:40933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:02:01.577826 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:02:01.578467 24854 tablet_replica.cc:330] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c: stopping tablet replica
I20240501 20:02:01.578976 24854 raft_consensus.cc:2229] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:01.579351 24854 raft_consensus.cc:2258] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:01.581001 24854 tablet_replica.cc:330] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c: stopping tablet replica
I20240501 20:02:01.581461 24854 raft_consensus.cc:2229] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:02:01.582062 24854 pending_rounds.cc:62] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c: Trying to abort 1 pending ops.
I20240501 20:02:01.582232 24854 pending_rounds.cc:68] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022975881895006208 op_type: NO_OP noop_request { }
I20240501 20:02:01.582504 24854 raft_consensus.cc:2866] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240501 20:02:01.582754 24854 raft_consensus.cc:2258] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:01.584704 24854 tablet_replica.cc:330] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c: stopping tablet replica
I20240501 20:02:01.585168 24854 raft_consensus.cc:2229] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:02:01.585664 24854 pending_rounds.cc:62] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c: Trying to abort 1 pending ops.
I20240501 20:02:01.585814 24854 pending_rounds.cc:68] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022975882800521216 op_type: NO_OP noop_request { }
I20240501 20:02:01.586064 24854 raft_consensus.cc:2866] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240501 20:02:01.586305 24854 raft_consensus.cc:2258] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:01.588003 24854 tablet_replica.cc:330] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c: stopping tablet replica
I20240501 20:02:01.588490 24854 raft_consensus.cc:2229] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:01.588814 24854 raft_consensus.cc:2258] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:01.609459 24854 tablet_server.cc:194] TabletServer@127.24.69.131:0 shutdown complete.
I20240501 20:02:01.620697 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:02:01.626502 29865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:01.627223 29866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:01.627938 29868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:02:01.628619 24854 server_base.cc:1034] running on GCE node
I20240501 20:02:01.629700 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:02:01.630013 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:02:01.630249 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593721630236 us; error 0 us; skew 500 ppm
I20240501 20:02:01.630859 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:02:01.638293 24854 webserver.cc:460] Webserver started at http://127.24.69.132:46137/ using document root <none> and password file <none>
I20240501 20:02:01.638881 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:02:01.639153 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:02:01.639494 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:02:01.640750 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-3-root/instance:
uuid: "43f052a8add942cf9d91f97be38da636"
format_stamp: "Formatted at 2024-05-01 20:02:01 on dist-test-slave-z8x3"
I20240501 20:02:01.645205 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20240501 20:02:01.648639 29873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:01.649487 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 20:02:01.649869 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-3-root
uuid: "43f052a8add942cf9d91f97be38da636"
format_stamp: "Formatted at 2024-05-01 20:02:01 on dist-test-slave-z8x3"
I20240501 20:02:01.650251 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:02:01.673296 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:02:01.674371 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:02:01.675868 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:02:01.678241 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:02:01.678547 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:01.678844 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:02:01.679083 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:01.714488 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.132:38453
I20240501 20:02:01.714572 29935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.132:38453 every 8 connection(s)
I20240501 20:02:01.728008 29936 heartbeater.cc:347] Connected to a master server at 127.24.69.190:35105
I20240501 20:02:01.728482 29936 heartbeater.cc:464] Registering TS with master...
I20240501 20:02:01.729271 29936 heartbeater.cc:511] Master 127.24.69.190:35105 requested a full tablet report, sending...
I20240501 20:02:01.731127 29537 ts_manager.cc:194] Registered new tserver with Master: 43f052a8add942cf9d91f97be38da636 (127.24.69.132:38453)
I20240501 20:02:01.732542 29537 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45088
W20240501 20:02:02.370973 29588 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.24.69.131:43209: connect: Connection refused (error 111)
W20240501 20:02:02.378512 29588 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.24.69.131:43209: connect: Connection refused (error 111)
W20240501 20:02:02.378926 29588 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.24.69.131:43209: connect: Connection refused (error 111)
W20240501 20:02:02.382939 29588 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44161: connect: Connection refused (error 111)
W20240501 20:02:02.383307 29588 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44161: connect: Connection refused (error 111)
W20240501 20:02:02.387101 29588 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44161: connect: Connection refused (error 111)
W20240501 20:02:02.387455 29588 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44161: connect: Connection refused (error 111)
I20240501 20:02:02.734851 29936 heartbeater.cc:503] Master 127.24.69.190:35105 was elected leader, sending a full tablet report...
W20240501 20:02:03.417472 29588 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.24.69.131:43209: connect: Connection refused (error 111)
W20240501 20:02:03.420331 29527 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.69.131:43209: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20240501 20:02:03.430539 29588 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.24.69.131:43209: connect: Connection refused (error 111)
W20240501 20:02:03.430819 29588 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.24.69.131:43209: connect: Connection refused (error 111)
W20240501 20:02:03.434749 29588 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44161: connect: Connection refused (error 111)
W20240501 20:02:03.434969 29588 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44161: connect: Connection refused (error 111)
W20240501 20:02:03.438746 29588 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44161: connect: Connection refused (error 111)
W20240501 20:02:03.438961 29588 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44161: connect: Connection refused (error 111)
W20240501 20:02:04.450146 29588 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.24.69.131:43209: connect: Connection refused (error 111)
W20240501 20:02:04.455343 29588 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.24.69.131:43209: connect: Connection refused (error 111)
W20240501 20:02:04.455567 29588 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.24.69.131:43209: connect: Connection refused (error 111)
W20240501 20:02:04.459363 29588 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44161: connect: Connection refused (error 111)
W20240501 20:02:04.459579 29588 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44161: connect: Connection refused (error 111)
W20240501 20:02:04.463400 29588 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44161: connect: Connection refused (error 111)
W20240501 20:02:04.463610 29588 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44161: connect: Connection refused (error 111)
W20240501 20:02:05.474596 29588 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44161: connect: Connection refused (error 111)
W20240501 20:02:05.480679 29588 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44161: connect: Connection refused (error 111)
W20240501 20:02:05.480962 29588 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.24.69.129:44161: connect: Connection refused (error 111)
W20240501 20:02:05.484833 29588 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.24.69.131:43209: connect: Connection refused (error 111)
W20240501 20:02:05.485100 29588 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.24.69.131:43209: connect: Connection refused (error 111)
W20240501 20:02:05.489033 29588 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.24.69.131:43209: connect: Connection refused (error 111)
W20240501 20:02:05.489277 29588 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.24.69.131:43209: connect: Connection refused (error 111)
W20240501 20:02:06.491433 29588 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 1e7eecb723634a5da85a7c4d19108130 not available for placement
W20240501 20:02:07.494109 29588 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 1e7eecb723634a5da85a7c4d19108130 not available for placement
I20240501 20:02:07.754530 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:02:07.763271 29953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:07.764067 29954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:07.765393 29956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:02:07.766880 24854 server_base.cc:1034] running on GCE node
I20240501 20:02:07.768193 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:02:07.768446 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:02:07.768612 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593727768594 us; error 0 us; skew 500 ppm
I20240501 20:02:07.769166 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:02:07.775554 24854 webserver.cc:460] Webserver started at http://127.24.69.129:37839/ using document root <none> and password file <none>
I20240501 20:02:07.776039 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:02:07.776225 24854 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240501 20:02:07.779983 24854 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:02:07.783125 29961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:07.783910 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:02:07.784191 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-0-root
uuid: "e93d3d097d994a1992554c82c6881b83"
format_stamp: "Formatted at 2024-05-01 20:01:59 on dist-test-slave-z8x3"
I20240501 20:02:07.784629 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:02:07.814013 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:02:07.815201 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:02:07.816653 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:02:07.820037 29968 ts_tablet_manager.cc:522] Loading tablet metadata (0/4 complete)
I20240501 20:02:07.841552 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240501 20:02:07.841815 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.023s user 0.001s sys 0.001s
I20240501 20:02:07.842063 24854 ts_tablet_manager.cc:574] Registering tablets (0/4 complete)
I20240501 20:02:07.846834 29968 tablet_bootstrap.cc:492] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83: Bootstrap starting.
I20240501 20:02:07.857708 24854 ts_tablet_manager.cc:590] Registered 4 tablets
I20240501 20:02:07.857950 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.016s user 0.015s sys 0.000s
I20240501 20:02:07.858891 29968 tablet_bootstrap.cc:492] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:02:07.859643 29968 tablet_bootstrap.cc:492] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83: Bootstrap complete.
I20240501 20:02:07.860133 29968 ts_tablet_manager.cc:1367] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20240501 20:02:07.862396 29968 raft_consensus.cc:348] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:07.863510 29968 raft_consensus.cc:729] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e93d3d097d994a1992554c82c6881b83, State: Initialized, Role: FOLLOWER
I20240501 20:02:07.864202 29968 consensus_queue.cc:260] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:07.866874 29968 ts_tablet_manager.cc:1398] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 20:02:07.867645 29968 tablet_bootstrap.cc:492] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83: Bootstrap starting.
I20240501 20:02:07.877943 29968 tablet_bootstrap.cc:492] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:02:07.878571 29968 tablet_bootstrap.cc:492] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83: Bootstrap complete.
I20240501 20:02:07.879072 29968 ts_tablet_manager.cc:1367] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240501 20:02:07.881155 29968 raft_consensus.cc:348] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:07.881675 29968 raft_consensus.cc:729] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e93d3d097d994a1992554c82c6881b83, State: Initialized, Role: FOLLOWER
I20240501 20:02:07.882292 29968 consensus_queue.cc:260] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:07.884071 29968 ts_tablet_manager.cc:1398] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 20:02:07.884737 29968 tablet_bootstrap.cc:492] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83: Bootstrap starting.
I20240501 20:02:07.895618 29968 tablet_bootstrap.cc:492] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:02:07.896376 29968 tablet_bootstrap.cc:492] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83: Bootstrap complete.
I20240501 20:02:07.896891 29968 ts_tablet_manager.cc:1367] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.002s
I20240501 20:02:07.899122 29968 raft_consensus.cc:348] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:07.899648 29968 raft_consensus.cc:729] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e93d3d097d994a1992554c82c6881b83, State: Initialized, Role: FOLLOWER
I20240501 20:02:07.900230 29968 consensus_queue.cc:260] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:07.902014 29968 ts_tablet_manager.cc:1398] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20240501 20:02:07.902645 29968 tablet_bootstrap.cc:492] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83: Bootstrap starting.
I20240501 20:02:07.914310 29968 tablet_bootstrap.cc:492] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:02:07.915189 29968 tablet_bootstrap.cc:492] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83: Bootstrap complete.
I20240501 20:02:07.915798 29968 ts_tablet_manager.cc:1367] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20240501 20:02:07.916718 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.129:44161
I20240501 20:02:07.916896 30029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.129:44161 every 8 connection(s)
I20240501 20:02:07.918145 29968 raft_consensus.cc:348] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:07.918840 29968 raft_consensus.cc:729] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e93d3d097d994a1992554c82c6881b83, State: Initialized, Role: FOLLOWER
I20240501 20:02:07.919423 29968 consensus_queue.cc:260] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
W20240501 20:02:07.920884 30031 tablet.cc:2369] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:02:07.921526 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 20:02:07.922166 29968 ts_tablet_manager.cc:1398] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
W20240501 20:02:07.930657 30034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:07.931020 30035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:02:07.936146 30030 heartbeater.cc:347] Connected to a master server at 127.24.69.190:35105
W20240501 20:02:07.936444 30037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:02:07.936582 30030 heartbeater.cc:464] Registering TS with master...
I20240501 20:02:07.937178 24854 server_base.cc:1034] running on GCE node
I20240501 20:02:07.937467 30030 heartbeater.cc:511] Master 127.24.69.190:35105 requested a full tablet report, sending...
I20240501 20:02:07.938329 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:02:07.938597 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:02:07.938798 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593727938780 us; error 0 us; skew 500 ppm
I20240501 20:02:07.939608 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:02:07.942174 29537 ts_manager.cc:194] Re-registered known tserver with Master: e93d3d097d994a1992554c82c6881b83 (127.24.69.129:44161)
I20240501 20:02:07.942461 24854 webserver.cc:460] Webserver started at http://127.24.69.130:44845/ using document root <none> and password file <none>
I20240501 20:02:07.942986 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:02:07.943140 24854 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240501 20:02:07.947086 24854 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20240501 20:02:07.947402 29537 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48138
I20240501 20:02:07.949784 30030 heartbeater.cc:503] Master 127.24.69.190:35105 was elected leader, sending a full tablet report...
I20240501 20:02:07.950413 30042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:07.951265 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:02:07.951529 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-1-root
uuid: "1e7eecb723634a5da85a7c4d19108130"
format_stamp: "Formatted at 2024-05-01 20:01:59 on dist-test-slave-z8x3"
I20240501 20:02:07.951900 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:02:07.988216 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:02:07.989388 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:02:07.990793 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:02:08.013526 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240501 20:02:08.013828 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.021s user 0.001s sys 0.000s
I20240501 20:02:08.018180 30049 tablet_bootstrap.cc:492] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130: Bootstrap starting.
I20240501 20:02:08.027997 30049 tablet_bootstrap.cc:492] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:02:08.028008 24854 ts_tablet_manager.cc:590] Registered 4 tablets
I20240501 20:02:08.028414 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.014s user 0.011s sys 0.003s
I20240501 20:02:08.028753 30049 tablet_bootstrap.cc:492] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130: Bootstrap complete.
I20240501 20:02:08.029389 30049 ts_tablet_manager.cc:1367] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240501 20:02:08.031260 30049 raft_consensus.cc:348] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:08.031759 30049 raft_consensus.cc:729] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e7eecb723634a5da85a7c4d19108130, State: Initialized, Role: FOLLOWER
I20240501 20:02:08.032358 30049 consensus_queue.cc:260] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:08.034477 30049 ts_tablet_manager.cc:1398] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 20:02:08.035074 30049 tablet_bootstrap.cc:492] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130: Bootstrap starting.
I20240501 20:02:08.045922 30049 tablet_bootstrap.cc:492] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:02:08.046819 30049 tablet_bootstrap.cc:492] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130: Bootstrap complete.
I20240501 20:02:08.047578 30049 ts_tablet_manager.cc:1367] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.001s
I20240501 20:02:08.049713 30049 raft_consensus.cc:348] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:08.050308 30049 raft_consensus.cc:729] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e7eecb723634a5da85a7c4d19108130, State: Initialized, Role: FOLLOWER
I20240501 20:02:08.051007 30049 consensus_queue.cc:260] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:08.052944 30049 ts_tablet_manager.cc:1398] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 20:02:08.053822 30049 tablet_bootstrap.cc:492] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130: Bootstrap starting.
I20240501 20:02:08.065145 30049 tablet_bootstrap.cc:492] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:02:08.065773 30049 tablet_bootstrap.cc:492] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130: Bootstrap complete.
I20240501 20:02:08.066282 30049 ts_tablet_manager.cc:1367] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20240501 20:02:08.068112 30049 raft_consensus.cc:348] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:08.068706 30049 raft_consensus.cc:729] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e7eecb723634a5da85a7c4d19108130, State: Initialized, Role: FOLLOWER
I20240501 20:02:08.069351 30049 consensus_queue.cc:260] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:08.071417 30049 ts_tablet_manager.cc:1398] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20240501 20:02:08.072010 30049 tablet_bootstrap.cc:492] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130: Bootstrap starting.
I20240501 20:02:08.083262 30049 tablet_bootstrap.cc:492] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:02:08.084190 30049 tablet_bootstrap.cc:492] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130: Bootstrap complete.
I20240501 20:02:08.085049 30049 ts_tablet_manager.cc:1367] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20240501 20:02:08.087589 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.130:40933
I20240501 20:02:08.087759 30110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.130:40933 every 8 connection(s)
I20240501 20:02:08.088132 30049 raft_consensus.cc:348] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:08.088913 30049 raft_consensus.cc:729] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e7eecb723634a5da85a7c4d19108130, State: Initialized, Role: FOLLOWER
I20240501 20:02:08.089637 30049 consensus_queue.cc:260] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:08.092082 30049 ts_tablet_manager.cc:1398] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240501 20:02:08.092911 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:02:08.101421 30114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:08.106349 30115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:02:08.106920 30111 heartbeater.cc:347] Connected to a master server at 127.24.69.190:35105
I20240501 20:02:08.107282 30111 heartbeater.cc:464] Registering TS with master...
I20240501 20:02:08.107990 30111 heartbeater.cc:511] Master 127.24.69.190:35105 requested a full tablet report, sending...
W20240501 20:02:08.110241 30117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:02:08.111378 24854 server_base.cc:1034] running on GCE node
I20240501 20:02:08.111986 29537 ts_manager.cc:194] Re-registered known tserver with Master: 1e7eecb723634a5da85a7c4d19108130 (127.24.69.130:40933)
I20240501 20:02:08.112231 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:02:08.112515 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:02:08.112669 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593728112652 us; error 0 us; skew 500 ppm
I20240501 20:02:08.113243 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:02:08.115844 24854 webserver.cc:460] Webserver started at http://127.24.69.131:38977/ using document root <none> and password file <none>
I20240501 20:02:08.116336 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:02:08.116510 24854 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240501 20:02:08.117094 29537 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48148
I20240501 20:02:08.119360 30111 heartbeater.cc:503] Master 127.24.69.190:35105 was elected leader, sending a full tablet report...
I20240501 20:02:08.120301 24854 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:02:08.123976 30122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:08.124914 24854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:02:08.125190 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-2-root
uuid: "5947681784a745eea2d6b0c97e234a4c"
format_stamp: "Formatted at 2024-05-01 20:01:59 on dist-test-slave-z8x3"
I20240501 20:02:08.125427 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1714593576104309-24854-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:02:08.141844 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:02:08.142968 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:02:08.144346 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:02:08.167271 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20240501 20:02:08.167536 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.021s user 0.001s sys 0.000s
I20240501 20:02:08.171751 30129 tablet_bootstrap.cc:492] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c: Bootstrap starting.
I20240501 20:02:08.181242 24854 ts_tablet_manager.cc:590] Registered 4 tablets
I20240501 20:02:08.181481 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.014s user 0.013s sys 0.000s
I20240501 20:02:08.182621 30129 tablet_bootstrap.cc:492] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:02:08.183265 30129 tablet_bootstrap.cc:492] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c: Bootstrap complete.
I20240501 20:02:08.183807 30129 ts_tablet_manager.cc:1367] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20240501 20:02:08.185750 30129 raft_consensus.cc:348] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:08.186340 30129 raft_consensus.cc:729] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5947681784a745eea2d6b0c97e234a4c, State: Initialized, Role: FOLLOWER
I20240501 20:02:08.186949 30129 consensus_queue.cc:260] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:08.189652 30129 ts_tablet_manager.cc:1398] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20240501 20:02:08.190485 30129 tablet_bootstrap.cc:492] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c: Bootstrap starting.
I20240501 20:02:08.202428 30129 tablet_bootstrap.cc:492] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:02:08.203186 30129 tablet_bootstrap.cc:492] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c: Bootstrap complete.
I20240501 20:02:08.203729 30129 ts_tablet_manager.cc:1367] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20240501 20:02:08.205778 30129 raft_consensus.cc:348] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:08.206480 30129 raft_consensus.cc:729] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5947681784a745eea2d6b0c97e234a4c, State: Initialized, Role: FOLLOWER
I20240501 20:02:08.207082 30129 consensus_queue.cc:260] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:08.208688 30129 ts_tablet_manager.cc:1398] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c: Time spent starting tablet: real 0.005s user 0.005s sys 0.003s
I20240501 20:02:08.209313 30129 tablet_bootstrap.cc:492] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c: Bootstrap starting.
I20240501 20:02:08.222476 30129 tablet_bootstrap.cc:492] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:02:08.223587 30129 tablet_bootstrap.cc:492] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c: Bootstrap complete.
I20240501 20:02:08.224308 30129 ts_tablet_manager.cc:1367] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20240501 20:02:08.227110 30129 raft_consensus.cc:348] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:08.228279 30129 raft_consensus.cc:729] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5947681784a745eea2d6b0c97e234a4c, State: Initialized, Role: FOLLOWER
I20240501 20:02:08.229167 30129 consensus_queue.cc:260] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:08.231842 30129 ts_tablet_manager.cc:1398] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240501 20:02:08.232781 30129 tablet_bootstrap.cc:492] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c: Bootstrap starting.
I20240501 20:02:08.236306 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.131:43209
I20240501 20:02:08.236481 30189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.131:43209 every 8 connection(s)
I20240501 20:02:08.248147 30129 tablet_bootstrap.cc:492] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:02:08.249253 30129 tablet_bootstrap.cc:492] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c: Bootstrap complete.
I20240501 20:02:08.250065 30129 ts_tablet_manager.cc:1367] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20240501 20:02:08.252637 30129 raft_consensus.cc:348] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:08.253553 30190 heartbeater.cc:347] Connected to a master server at 127.24.69.190:35105
I20240501 20:02:08.253567 30129 raft_consensus.cc:729] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5947681784a745eea2d6b0c97e234a4c, State: Initialized, Role: FOLLOWER
I20240501 20:02:08.254312 30190 heartbeater.cc:464] Registering TS with master...
I20240501 20:02:08.255380 30190 heartbeater.cc:511] Master 127.24.69.190:35105 requested a full tablet report, sending...
I20240501 20:02:08.255014 30129 consensus_queue.cc:260] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:08.257786 30129 ts_tablet_manager.cc:1398] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240501 20:02:08.259683 29537 ts_manager.cc:194] Re-registered known tserver with Master: 5947681784a745eea2d6b0c97e234a4c (127.24.69.131:43209)
I20240501 20:02:08.263904 29537 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48160
I20240501 20:02:08.266192 30190 heartbeater.cc:503] Master 127.24.69.190:35105 was elected leader, sending a full tablet report...
W20240501 20:02:08.514767 29588 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica e93d3d097d994a1992554c82c6881b83 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } } }
W20240501 20:02:08.518347 29588 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ed6ad07388d449cf8a8504a00f60dbd8, TS 5947681784a745eea2d6b0c97e234a4c -> TS 43f052a8add942cf9d91f97be38da636 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
W20240501 20:02:08.518599 29588 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet ed6ad07388d449cf8a8504a00f60dbd8, TS 5947681784a745eea2d6b0c97e234a4c -> TS 43f052a8add942cf9d91f97be38da636 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
W20240501 20:02:08.530345 29588 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 96a2762d729f41a3a34b013bf76d4702, TS e93d3d097d994a1992554c82c6881b83 -> TS 43f052a8add942cf9d91f97be38da636 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
W20240501 20:02:08.530565 29588 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 96a2762d729f41a3a34b013bf76d4702, TS e93d3d097d994a1992554c82c6881b83 -> TS 43f052a8add942cf9d91f97be38da636 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
W20240501 20:02:08.533378 29588 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 7b3d39a2ed344577b6712279908a54ff, TS 1e7eecb723634a5da85a7c4d19108130 -> TS 43f052a8add942cf9d91f97be38da636 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
W20240501 20:02:08.533581 29588 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 7b3d39a2ed344577b6712279908a54ff, TS 1e7eecb723634a5da85a7c4d19108130 -> TS 43f052a8add942cf9d91f97be38da636 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:09.294909 30197 raft_consensus.cc:482] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:09.295437 30197 raft_consensus.cc:504] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:09.297745 30197 leader_election.cc:288] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e7eecb723634a5da85a7c4d19108130 (127.24.69.130:40933), 5947681784a745eea2d6b0c97e234a4c (127.24.69.131:43209)
I20240501 20:02:09.308502 30084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a2762d729f41a3a34b013bf76d4702" candidate_uuid: "e93d3d097d994a1992554c82c6881b83" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e7eecb723634a5da85a7c4d19108130" is_pre_election: true
I20240501 20:02:09.309022 30164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a2762d729f41a3a34b013bf76d4702" candidate_uuid: "e93d3d097d994a1992554c82c6881b83" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5947681784a745eea2d6b0c97e234a4c" is_pre_election: true
I20240501 20:02:09.309306 30084 raft_consensus.cc:2454] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e93d3d097d994a1992554c82c6881b83 in term 1.
I20240501 20:02:09.309828 30164 raft_consensus.cc:2396] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e93d3d097d994a1992554c82c6881b83 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240501 20:02:09.310411 29963 leader_election.cc:302] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e7eecb723634a5da85a7c4d19108130, e93d3d097d994a1992554c82c6881b83; no voters:
I20240501 20:02:09.311257 30197 raft_consensus.cc:2781] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:02:09.311592 30197 raft_consensus.cc:482] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:02:09.311859 30197 raft_consensus.cc:3037] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:02:09.316793 30197 raft_consensus.cc:504] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:09.318234 30197 leader_election.cc:288] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [CANDIDATE]: Term 2 election: Requested vote from peers 1e7eecb723634a5da85a7c4d19108130 (127.24.69.130:40933), 5947681784a745eea2d6b0c97e234a4c (127.24.69.131:43209)
I20240501 20:02:09.318970 30084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a2762d729f41a3a34b013bf76d4702" candidate_uuid: "e93d3d097d994a1992554c82c6881b83" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e7eecb723634a5da85a7c4d19108130"
I20240501 20:02:09.319159 30164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a2762d729f41a3a34b013bf76d4702" candidate_uuid: "e93d3d097d994a1992554c82c6881b83" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5947681784a745eea2d6b0c97e234a4c"
I20240501 20:02:09.319574 30084 raft_consensus.cc:3037] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:02:09.319705 30164 raft_consensus.cc:3037] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:02:09.326687 30084 raft_consensus.cc:2454] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e93d3d097d994a1992554c82c6881b83 in term 2.
I20240501 20:02:09.327015 30164 raft_consensus.cc:2396] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e93d3d097d994a1992554c82c6881b83 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240501 20:02:09.327775 29963 leader_election.cc:302] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e7eecb723634a5da85a7c4d19108130, e93d3d097d994a1992554c82c6881b83; no voters:
I20240501 20:02:09.328418 30197 raft_consensus.cc:2781] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:02:09.329423 30197 raft_consensus.cc:686] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [term 2 LEADER]: Becoming Leader. State: Replica: e93d3d097d994a1992554c82c6881b83, State: Running, Role: LEADER
I20240501 20:02:09.330193 30197 consensus_queue.cc:237] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:09.336366 29537 catalog_manager.cc:5484] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 reported cstate change: term changed from 1 to 2, leader changed from 5947681784a745eea2d6b0c97e234a4c (127.24.69.131) to e93d3d097d994a1992554c82c6881b83 (127.24.69.129). New cstate: current_term: 2 leader_uuid: "e93d3d097d994a1992554c82c6881b83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } health_report { overall_health: UNKNOWN } } }
I20240501 20:02:09.424546 30208 raft_consensus.cc:482] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:09.424978 30208 raft_consensus.cc:504] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:09.426873 30208 leader_election.cc:288] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e93d3d097d994a1992554c82c6881b83 (127.24.69.129:44161), 5947681784a745eea2d6b0c97e234a4c (127.24.69.131:43209)
I20240501 20:02:09.438760 30164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed6ad07388d449cf8a8504a00f60dbd8" candidate_uuid: "1e7eecb723634a5da85a7c4d19108130" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5947681784a745eea2d6b0c97e234a4c" is_pre_election: true
I20240501 20:02:09.439399 30164 raft_consensus.cc:2454] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e7eecb723634a5da85a7c4d19108130 in term 1.
I20240501 20:02:09.439329 30003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed6ad07388d449cf8a8504a00f60dbd8" candidate_uuid: "1e7eecb723634a5da85a7c4d19108130" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e93d3d097d994a1992554c82c6881b83" is_pre_election: true
I20240501 20:02:09.440374 30003 raft_consensus.cc:2396] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e7eecb723634a5da85a7c4d19108130 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240501 20:02:09.440450 30046 leader_election.cc:302] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e7eecb723634a5da85a7c4d19108130, 5947681784a745eea2d6b0c97e234a4c; no voters:
I20240501 20:02:09.441154 30208 raft_consensus.cc:2781] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:02:09.441512 30208 raft_consensus.cc:482] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:02:09.441908 30208 raft_consensus.cc:3037] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:02:09.446893 30208 raft_consensus.cc:504] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:09.448421 30208 leader_election.cc:288] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [CANDIDATE]: Term 2 election: Requested vote from peers e93d3d097d994a1992554c82c6881b83 (127.24.69.129:44161), 5947681784a745eea2d6b0c97e234a4c (127.24.69.131:43209)
I20240501 20:02:09.449191 30003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed6ad07388d449cf8a8504a00f60dbd8" candidate_uuid: "1e7eecb723634a5da85a7c4d19108130" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e93d3d097d994a1992554c82c6881b83"
I20240501 20:02:09.449368 30164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed6ad07388d449cf8a8504a00f60dbd8" candidate_uuid: "1e7eecb723634a5da85a7c4d19108130" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5947681784a745eea2d6b0c97e234a4c"
I20240501 20:02:09.449659 30003 raft_consensus.cc:3037] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:02:09.449885 30164 raft_consensus.cc:3037] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:02:09.454627 30164 raft_consensus.cc:2454] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e7eecb723634a5da85a7c4d19108130 in term 2.
I20240501 20:02:09.455173 30003 raft_consensus.cc:2396] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e7eecb723634a5da85a7c4d19108130 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240501 20:02:09.455533 30046 leader_election.cc:302] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e7eecb723634a5da85a7c4d19108130, 5947681784a745eea2d6b0c97e234a4c; no voters:
I20240501 20:02:09.456329 30208 raft_consensus.cc:2781] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:02:09.457365 30208 raft_consensus.cc:686] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 2 LEADER]: Becoming Leader. State: Replica: 1e7eecb723634a5da85a7c4d19108130, State: Running, Role: LEADER
I20240501 20:02:09.457980 30208 consensus_queue.cc:237] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:09.464155 29537 catalog_manager.cc:5484] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 reported cstate change: term changed from 1 to 2, leader changed from e93d3d097d994a1992554c82c6881b83 (127.24.69.129) to 1e7eecb723634a5da85a7c4d19108130 (127.24.69.130). New cstate: current_term: 2 leader_uuid: "1e7eecb723634a5da85a7c4d19108130" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } health_report { overall_health: UNKNOWN } } }
I20240501 20:02:09.514467 30208 raft_consensus.cc:482] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:09.514978 30208 raft_consensus.cc:504] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:09.516666 30208 leader_election.cc:288] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e93d3d097d994a1992554c82c6881b83 (127.24.69.129:44161), 5947681784a745eea2d6b0c97e234a4c (127.24.69.131:43209)
I20240501 20:02:09.517485 30003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd88640130a342d899a02a89c3a9fcfa" candidate_uuid: "1e7eecb723634a5da85a7c4d19108130" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e93d3d097d994a1992554c82c6881b83" is_pre_election: true
I20240501 20:02:09.517679 30164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd88640130a342d899a02a89c3a9fcfa" candidate_uuid: "1e7eecb723634a5da85a7c4d19108130" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5947681784a745eea2d6b0c97e234a4c" is_pre_election: true
I20240501 20:02:09.518054 30003 raft_consensus.cc:2454] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e7eecb723634a5da85a7c4d19108130 in term 1.
I20240501 20:02:09.518456 30164 raft_consensus.cc:2396] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e7eecb723634a5da85a7c4d19108130 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240501 20:02:09.519093 30044 leader_election.cc:302] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e7eecb723634a5da85a7c4d19108130, e93d3d097d994a1992554c82c6881b83; no voters:
I20240501 20:02:09.519758 30208 raft_consensus.cc:2781] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:02:09.520090 30208 raft_consensus.cc:482] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:02:09.520328 30208 raft_consensus.cc:3037] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:02:09.524693 30208 raft_consensus.cc:504] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:09.526234 30208 leader_election.cc:288] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [CANDIDATE]: Term 2 election: Requested vote from peers e93d3d097d994a1992554c82c6881b83 (127.24.69.129:44161), 5947681784a745eea2d6b0c97e234a4c (127.24.69.131:43209)
I20240501 20:02:09.527210 30164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd88640130a342d899a02a89c3a9fcfa" candidate_uuid: "1e7eecb723634a5da85a7c4d19108130" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5947681784a745eea2d6b0c97e234a4c"
I20240501 20:02:09.527088 30003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd88640130a342d899a02a89c3a9fcfa" candidate_uuid: "1e7eecb723634a5da85a7c4d19108130" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e93d3d097d994a1992554c82c6881b83"
I20240501 20:02:09.527806 30164 raft_consensus.cc:3037] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:02:09.527909 30003 raft_consensus.cc:3037] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:02:09.532426 30164 raft_consensus.cc:2396] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e7eecb723634a5da85a7c4d19108130 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240501 20:02:09.532574 30003 raft_consensus.cc:2454] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e7eecb723634a5da85a7c4d19108130 in term 2.
I20240501 20:02:09.533454 30044 leader_election.cc:302] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e7eecb723634a5da85a7c4d19108130, e93d3d097d994a1992554c82c6881b83; no voters: 5947681784a745eea2d6b0c97e234a4c
I20240501 20:02:09.534062 30208 raft_consensus.cc:2781] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:02:09.534514 30208 raft_consensus.cc:686] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 2 LEADER]: Becoming Leader. State: Replica: 1e7eecb723634a5da85a7c4d19108130, State: Running, Role: LEADER
I20240501 20:02:09.535172 30208 consensus_queue.cc:237] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:09.541347 29537 catalog_manager.cc:5484] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 reported cstate change: term changed from 1 to 2, leader changed from 5947681784a745eea2d6b0c97e234a4c (127.24.69.131) to 1e7eecb723634a5da85a7c4d19108130 (127.24.69.130). New cstate: current_term: 2 leader_uuid: "1e7eecb723634a5da85a7c4d19108130" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } health_report { overall_health: UNKNOWN } } }
I20240501 20:02:09.547178 30197 raft_consensus.cc:482] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:09.547686 30197 raft_consensus.cc:504] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:09.549327 30197 leader_election.cc:288] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e7eecb723634a5da85a7c4d19108130 (127.24.69.130:40933), 5947681784a745eea2d6b0c97e234a4c (127.24.69.131:43209)
I20240501 20:02:09.550153 30084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3d39a2ed344577b6712279908a54ff" candidate_uuid: "e93d3d097d994a1992554c82c6881b83" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e7eecb723634a5da85a7c4d19108130" is_pre_election: true
I20240501 20:02:09.550817 30084 raft_consensus.cc:2454] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e93d3d097d994a1992554c82c6881b83 in term 1.
I20240501 20:02:09.550942 30164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3d39a2ed344577b6712279908a54ff" candidate_uuid: "e93d3d097d994a1992554c82c6881b83" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5947681784a745eea2d6b0c97e234a4c" is_pre_election: true
I20240501 20:02:09.551669 30164 raft_consensus.cc:2454] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e93d3d097d994a1992554c82c6881b83 in term 1.
I20240501 20:02:09.552042 29963 leader_election.cc:302] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e7eecb723634a5da85a7c4d19108130, e93d3d097d994a1992554c82c6881b83; no voters:
I20240501 20:02:09.553931 30197 raft_consensus.cc:2781] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:02:09.554220 30197 raft_consensus.cc:482] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:02:09.554517 30197 raft_consensus.cc:3037] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:02:09.554793 29588 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20240501 20:02:09.558518 29588 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ed6ad07388d449cf8a8504a00f60dbd8, TS e93d3d097d994a1992554c82c6881b83 -> TS 43f052a8add942cf9d91f97be38da636 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
W20240501 20:02:09.558830 29588 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet ed6ad07388d449cf8a8504a00f60dbd8, TS e93d3d097d994a1992554c82c6881b83 -> TS 43f052a8add942cf9d91f97be38da636 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:09.560369 30197 raft_consensus.cc:504] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:09.561058 30216 raft_consensus.cc:482] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:09.561512 30216 raft_consensus.cc:504] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
W20240501 20:02:09.562712 29588 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 96a2762d729f41a3a34b013bf76d4702, TS 1e7eecb723634a5da85a7c4d19108130 -> TS 43f052a8add942cf9d91f97be38da636 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
W20240501 20:02:09.563015 29588 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 96a2762d729f41a3a34b013bf76d4702, TS 1e7eecb723634a5da85a7c4d19108130 -> TS 43f052a8add942cf9d91f97be38da636 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:09.563175 30084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3d39a2ed344577b6712279908a54ff" candidate_uuid: "e93d3d097d994a1992554c82c6881b83" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e7eecb723634a5da85a7c4d19108130"
I20240501 20:02:09.564028 30216 leader_election.cc:288] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e93d3d097d994a1992554c82c6881b83 (127.24.69.129:44161), 1e7eecb723634a5da85a7c4d19108130 (127.24.69.130:40933)
I20240501 20:02:09.563751 30084 raft_consensus.cc:3037] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:02:09.568606 29588 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 7b3d39a2ed344577b6712279908a54ff, TS 5947681784a745eea2d6b0c97e234a4c -> TS 43f052a8add942cf9d91f97be38da636 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
W20240501 20:02:09.568926 29588 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 7b3d39a2ed344577b6712279908a54ff, TS 5947681784a745eea2d6b0c97e234a4c -> TS 43f052a8add942cf9d91f97be38da636 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:09.569087 30197 leader_election.cc:288] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [CANDIDATE]: Term 2 election: Requested vote from peers 1e7eecb723634a5da85a7c4d19108130 (127.24.69.130:40933), 5947681784a745eea2d6b0c97e234a4c (127.24.69.131:43209)
I20240501 20:02:09.571642 30084 raft_consensus.cc:2454] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e93d3d097d994a1992554c82c6881b83 in term 2.
I20240501 20:02:09.563485 30164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3d39a2ed344577b6712279908a54ff" candidate_uuid: "e93d3d097d994a1992554c82c6881b83" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5947681784a745eea2d6b0c97e234a4c"
I20240501 20:02:09.572638 29963 leader_election.cc:302] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e7eecb723634a5da85a7c4d19108130, e93d3d097d994a1992554c82c6881b83; no voters:
I20240501 20:02:09.573247 30197 raft_consensus.cc:2781] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:02:09.573381 30164 raft_consensus.cc:3037] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:02:09.573772 30197 raft_consensus.cc:686] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [term 2 LEADER]: Becoming Leader. State: Replica: e93d3d097d994a1992554c82c6881b83, State: Running, Role: LEADER
I20240501 20:02:09.574582 30197 consensus_queue.cc:237] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:09.583119 30164 raft_consensus.cc:2454] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e93d3d097d994a1992554c82c6881b83 in term 2.
I20240501 20:02:09.583703 30003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd88640130a342d899a02a89c3a9fcfa" candidate_uuid: "5947681784a745eea2d6b0c97e234a4c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e93d3d097d994a1992554c82c6881b83" is_pre_election: true
I20240501 20:02:09.584510 30003 raft_consensus.cc:2454] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5947681784a745eea2d6b0c97e234a4c in term 2.
I20240501 20:02:09.585640 30124 leader_election.cc:302] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5947681784a745eea2d6b0c97e234a4c, e93d3d097d994a1992554c82c6881b83; no voters:
I20240501 20:02:09.586546 30216 raft_consensus.cc:2781] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:02:09.586926 30216 raft_consensus.cc:482] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:02:09.587249 30216 raft_consensus.cc:3037] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:02:09.588001 29537 catalog_manager.cc:5484] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "e93d3d097d994a1992554c82c6881b83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } health_report { overall_health: UNKNOWN } } }
I20240501 20:02:09.591012 30197 raft_consensus.cc:482] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:09.591399 30197 raft_consensus.cc:504] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:09.592029 30084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd88640130a342d899a02a89c3a9fcfa" candidate_uuid: "5947681784a745eea2d6b0c97e234a4c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e7eecb723634a5da85a7c4d19108130" is_pre_election: true
I20240501 20:02:09.593976 30084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed6ad07388d449cf8a8504a00f60dbd8" candidate_uuid: "e93d3d097d994a1992554c82c6881b83" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e7eecb723634a5da85a7c4d19108130" is_pre_election: true
I20240501 20:02:09.595180 30216 raft_consensus.cc:504] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:09.595573 30197 leader_election.cc:288] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e7eecb723634a5da85a7c4d19108130 (127.24.69.130:40933), 5947681784a745eea2d6b0c97e234a4c (127.24.69.131:43209)
I20240501 20:02:09.596617 30164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed6ad07388d449cf8a8504a00f60dbd8" candidate_uuid: "e93d3d097d994a1992554c82c6881b83" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5947681784a745eea2d6b0c97e234a4c" is_pre_election: true
I20240501 20:02:09.597115 30164 raft_consensus.cc:2454] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e93d3d097d994a1992554c82c6881b83 in term 2.
I20240501 20:02:09.598181 30216 leader_election.cc:288] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [CANDIDATE]: Term 3 election: Requested vote from peers e93d3d097d994a1992554c82c6881b83 (127.24.69.129:44161), 1e7eecb723634a5da85a7c4d19108130 (127.24.69.130:40933)
I20240501 20:02:09.598402 30003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd88640130a342d899a02a89c3a9fcfa" candidate_uuid: "5947681784a745eea2d6b0c97e234a4c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e93d3d097d994a1992554c82c6881b83"
I20240501 20:02:09.598627 30084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd88640130a342d899a02a89c3a9fcfa" candidate_uuid: "5947681784a745eea2d6b0c97e234a4c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e7eecb723634a5da85a7c4d19108130"
I20240501 20:02:09.598980 30003 raft_consensus.cc:3037] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:02:09.598968 29965 leader_election.cc:302] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5947681784a745eea2d6b0c97e234a4c, e93d3d097d994a1992554c82c6881b83; no voters: 1e7eecb723634a5da85a7c4d19108130
I20240501 20:02:09.600425 30197 raft_consensus.cc:2781] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:02:09.600787 30197 raft_consensus.cc:482] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:02:09.601119 30197 raft_consensus.cc:3037] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:02:09.604017 30003 raft_consensus.cc:2454] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5947681784a745eea2d6b0c97e234a4c in term 3.
I20240501 20:02:09.604914 30124 leader_election.cc:302] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5947681784a745eea2d6b0c97e234a4c, e93d3d097d994a1992554c82c6881b83; no voters: 1e7eecb723634a5da85a7c4d19108130
I20240501 20:02:09.605546 30216 raft_consensus.cc:2781] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:02:09.605974 30197 raft_consensus.cc:504] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:09.606791 30216 raft_consensus.cc:686] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 3 LEADER]: Becoming Leader. State: Replica: 5947681784a745eea2d6b0c97e234a4c, State: Running, Role: LEADER
I20240501 20:02:09.607677 30216 consensus_queue.cc:237] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:09.608733 30084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed6ad07388d449cf8a8504a00f60dbd8" candidate_uuid: "e93d3d097d994a1992554c82c6881b83" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1e7eecb723634a5da85a7c4d19108130"
I20240501 20:02:09.609194 30197 leader_election.cc:288] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [CANDIDATE]: Term 3 election: Requested vote from peers 1e7eecb723634a5da85a7c4d19108130 (127.24.69.130:40933), 5947681784a745eea2d6b0c97e234a4c (127.24.69.131:43209)
I20240501 20:02:09.609433 30164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed6ad07388d449cf8a8504a00f60dbd8" candidate_uuid: "e93d3d097d994a1992554c82c6881b83" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5947681784a745eea2d6b0c97e234a4c"
I20240501 20:02:09.610059 30164 raft_consensus.cc:3037] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:02:09.615577 30164 raft_consensus.cc:2454] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e93d3d097d994a1992554c82c6881b83 in term 3.
I20240501 20:02:09.616431 29965 leader_election.cc:302] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5947681784a745eea2d6b0c97e234a4c, e93d3d097d994a1992554c82c6881b83; no voters: 1e7eecb723634a5da85a7c4d19108130
I20240501 20:02:09.616182 29537 catalog_manager.cc:5484] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c reported cstate change: term changed from 2 to 3, leader changed from 1e7eecb723634a5da85a7c4d19108130 (127.24.69.130) to 5947681784a745eea2d6b0c97e234a4c (127.24.69.131). New cstate: current_term: 3 leader_uuid: "5947681784a745eea2d6b0c97e234a4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } health_report { overall_health: HEALTHY } } }
I20240501 20:02:09.617089 30197 raft_consensus.cc:2781] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:02:09.617569 30197 raft_consensus.cc:686] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 3 LEADER]: Becoming Leader. State: Replica: e93d3d097d994a1992554c82c6881b83, State: Running, Role: LEADER
I20240501 20:02:09.618325 30197 consensus_queue.cc:237] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:09.625656 29539 catalog_manager.cc:5484] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 reported cstate change: term changed from 2 to 3, leader changed from 1e7eecb723634a5da85a7c4d19108130 (127.24.69.130) to e93d3d097d994a1992554c82c6881b83 (127.24.69.129). New cstate: current_term: 3 leader_uuid: "e93d3d097d994a1992554c82c6881b83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } health_report { overall_health: UNKNOWN } } }
I20240501 20:02:09.712577 30164 raft_consensus.cc:1261] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [term 2 FOLLOWER]: Refusing update from remote peer e93d3d097d994a1992554c82c6881b83: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20240501 20:02:09.712954 30164 pending_rounds.cc:76] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c: Aborting all ops after (but not including) 0
I20240501 20:02:09.713165 30164 pending_rounds.cc:97] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20240501 20:02:09.713420 30164 raft_consensus.cc:2866] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:02:09.714676 30197 consensus_queue.cc:1035] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:02:09.725857 30084 raft_consensus.cc:1261] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130 [term 2 FOLLOWER]: Refusing update from remote peer e93d3d097d994a1992554c82c6881b83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240501 20:02:09.727242 30197 consensus_queue.cc:1035] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:02:09.881147 30164 raft_consensus.cc:1226] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c [term 3 FOLLOWER]: Rejecting Update request from peer 1e7eecb723634a5da85a7c4d19108130 for earlier term 2. Current term is 3. Ops: []
I20240501 20:02:09.882320 30208 consensus_queue.cc:1046] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:02:09.883054 30212 raft_consensus.cc:3032] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:02:09.883342 30212 raft_consensus.cc:729] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e7eecb723634a5da85a7c4d19108130, State: Running, Role: LEADER
I20240501 20:02:09.883962 30212 consensus_queue.cc:260] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:09.884841 30212 raft_consensus.cc:3037] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:02:09.970122 30003 raft_consensus.cc:1226] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83 [term 3 FOLLOWER]: Rejecting Update request from peer 1e7eecb723634a5da85a7c4d19108130 for earlier term 2. Current term is 3. Ops: []
I20240501 20:02:09.971441 30212 consensus_queue.cc:1046] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:02:09.972077 30208 raft_consensus.cc:3032] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:02:09.972368 30208 raft_consensus.cc:729] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e7eecb723634a5da85a7c4d19108130, State: Running, Role: LEADER
I20240501 20:02:09.972982 30208 consensus_queue.cc:260] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } }
I20240501 20:02:09.973811 30208 raft_consensus.cc:3037] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:02:09.991770 30003 raft_consensus.cc:1261] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83 [term 3 FOLLOWER]: Refusing update from remote peer 5947681784a745eea2d6b0c97e234a4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:02:09.993063 30216 consensus_queue.cc:1035] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:02:10.003794 30084 raft_consensus.cc:1261] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 3 FOLLOWER]: Refusing update from remote peer 5947681784a745eea2d6b0c97e234a4c: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:02:10.005328 30229 consensus_queue.cc:1035] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:02:10.006534 30229 consensus_queue.cc:1225] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [LEADER]: Peer 1e7eecb723634a5da85a7c4d19108130 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240501 20:02:10.009443 30084 pending_rounds.cc:76] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130: Aborting all ops after (but not including) 0
I20240501 20:02:10.009752 30084 pending_rounds.cc:97] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20240501 20:02:10.010012 30084 raft_consensus.cc:2866] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:02:10.020483 30084 raft_consensus.cc:1261] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130 [term 2 FOLLOWER]: Refusing update from remote peer e93d3d097d994a1992554c82c6881b83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:02:10.021569 30197 consensus_queue.cc:1035] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:02:10.024245 30164 raft_consensus.cc:1261] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c [term 2 FOLLOWER]: Refusing update from remote peer e93d3d097d994a1992554c82c6881b83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:02:10.025684 30202 consensus_queue.cc:1035] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:02:10.120146 30164 raft_consensus.cc:1261] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c [term 3 FOLLOWER]: Refusing update from remote peer e93d3d097d994a1992554c82c6881b83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:02:10.121301 30202 consensus_queue.cc:1035] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:02:10.132249 30084 raft_consensus.cc:1261] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 3 FOLLOWER]: Refusing update from remote peer e93d3d097d994a1992554c82c6881b83: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:02:10.134514 30232 consensus_queue.cc:1035] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:02:10.135325 30232 consensus_queue.cc:1225] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [LEADER]: Peer 1e7eecb723634a5da85a7c4d19108130 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240501 20:02:10.138072 30084 pending_rounds.cc:76] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130: Aborting all ops after (but not including) 0
I20240501 20:02:10.138329 30084 pending_rounds.cc:97] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20240501 20:02:10.138593 30084 raft_consensus.cc:2866] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:02:10.578294 30164 consensus_queue.cc:237] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } attrs { replace: true } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } } peers { permanent_uuid: "43f052a8add942cf9d91f97be38da636" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 38453 } attrs { promote: true } }
I20240501 20:02:10.583490 30083 raft_consensus.cc:1261] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 3 FOLLOWER]: Refusing update from remote peer 5947681784a745eea2d6b0c97e234a4c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:02:10.584708 30003 raft_consensus.cc:1261] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83 [term 3 FOLLOWER]: Refusing update from remote peer 5947681784a745eea2d6b0c97e234a4c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:02:10.585098 30229 consensus_queue.cc:1035] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:02:10.586931 30220 consensus_queue.cc:1035] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:02:10.593123 30229 raft_consensus.cc:2932] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 43f052a8add942cf9d91f97be38da636 (127.24.69.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } attrs { replace: true } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } } peers { permanent_uuid: "43f052a8add942cf9d91f97be38da636" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 38453 } attrs { promote: true } } }
I20240501 20:02:10.595561 30003 raft_consensus.cc:2932] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 43f052a8add942cf9d91f97be38da636 (127.24.69.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } attrs { replace: true } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } } peers { permanent_uuid: "43f052a8add942cf9d91f97be38da636" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 38453 } attrs { promote: true } } }
W20240501 20:02:10.597569 30123 consensus_peers.cc:485] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c -> Peer 43f052a8add942cf9d91f97be38da636 (127.24.69.132:38453): Couldn't send request to peer 43f052a8add942cf9d91f97be38da636. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd88640130a342d899a02a89c3a9fcfa. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:02:10.596607 30083 raft_consensus.cc:2932] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 43f052a8add942cf9d91f97be38da636 (127.24.69.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } attrs { replace: true } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } } peers { permanent_uuid: "43f052a8add942cf9d91f97be38da636" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 38453 } attrs { promote: true } } }
I20240501 20:02:10.607023 30002 consensus_queue.cc:237] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } attrs { replace: true } } peers { permanent_uuid: "43f052a8add942cf9d91f97be38da636" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 38453 } attrs { promote: true } }
I20240501 20:02:10.609479 29537 catalog_manager.cc:5484] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c reported cstate change: config changed from index -1 to 3, NON_VOTER 43f052a8add942cf9d91f97be38da636 (127.24.69.132) added. New cstate: current_term: 3 leader_uuid: "5947681784a745eea2d6b0c97e234a4c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43f052a8add942cf9d91f97be38da636" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 38453 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:02:10.614087 30164 raft_consensus.cc:1261] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c [term 3 FOLLOWER]: Refusing update from remote peer e93d3d097d994a1992554c82c6881b83: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:02:10.616351 30232 consensus_queue.cc:1035] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:02:10.616727 30083 raft_consensus.cc:1261] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 3 FOLLOWER]: Refusing update from remote peer e93d3d097d994a1992554c82c6881b83: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:02:10.618256 30232 consensus_queue.cc:1035] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:02:10.624011 30202 raft_consensus.cc:2932] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 43f052a8add942cf9d91f97be38da636 (127.24.69.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } attrs { replace: true } } peers { permanent_uuid: "43f052a8add942cf9d91f97be38da636" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 38453 } attrs { promote: true } } }
W20240501 20:02:10.627802 29962 consensus_peers.cc:485] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 -> Peer 43f052a8add942cf9d91f97be38da636 (127.24.69.132:38453): Couldn't send request to peer 43f052a8add942cf9d91f97be38da636. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed6ad07388d449cf8a8504a00f60dbd8. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:02:10.626902 30164 raft_consensus.cc:2932] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 43f052a8add942cf9d91f97be38da636 (127.24.69.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } attrs { replace: true } } peers { permanent_uuid: "43f052a8add942cf9d91f97be38da636" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 38453 } attrs { promote: true } } }
I20240501 20:02:10.628070 30083 raft_consensus.cc:2932] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 43f052a8add942cf9d91f97be38da636 (127.24.69.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } attrs { replace: true } } peers { permanent_uuid: "43f052a8add942cf9d91f97be38da636" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 38453 } attrs { promote: true } } }
I20240501 20:02:10.642513 30002 consensus_queue.cc:237] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } attrs { replace: true } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } } peers { permanent_uuid: "43f052a8add942cf9d91f97be38da636" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 38453 } attrs { promote: true } }
I20240501 20:02:10.642261 29537 catalog_manager.cc:5484] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 reported cstate change: config changed from index -1 to 3, NON_VOTER 43f052a8add942cf9d91f97be38da636 (127.24.69.132) added. New cstate: current_term: 3 leader_uuid: "e93d3d097d994a1992554c82c6881b83" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43f052a8add942cf9d91f97be38da636" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 38453 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:02:10.648754 30083 raft_consensus.cc:1261] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130 [term 2 FOLLOWER]: Refusing update from remote peer e93d3d097d994a1992554c82c6881b83: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240501 20:02:10.648953 29962 consensus_peers.cc:485] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 -> Peer 43f052a8add942cf9d91f97be38da636 (127.24.69.132:38453): Couldn't send request to peer 43f052a8add942cf9d91f97be38da636. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96a2762d729f41a3a34b013bf76d4702. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:02:10.649021 30164 raft_consensus.cc:1261] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [term 2 FOLLOWER]: Refusing update from remote peer e93d3d097d994a1992554c82c6881b83: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:02:10.650242 30197 consensus_queue.cc:1035] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:02:10.651189 30232 consensus_queue.cc:1035] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:02:10.658876 30202 raft_consensus.cc:2932] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 43f052a8add942cf9d91f97be38da636 (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } attrs { replace: true } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } } peers { permanent_uuid: "43f052a8add942cf9d91f97be38da636" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 38453 } attrs { promote: true } } }
I20240501 20:02:10.660161 30083 raft_consensus.cc:2932] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 43f052a8add942cf9d91f97be38da636 (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } attrs { replace: true } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } } peers { permanent_uuid: "43f052a8add942cf9d91f97be38da636" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 38453 } attrs { promote: true } } }
I20240501 20:02:10.660166 30164 raft_consensus.cc:2932] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 43f052a8add942cf9d91f97be38da636 (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } attrs { replace: true } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } } peers { permanent_uuid: "43f052a8add942cf9d91f97be38da636" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 38453 } attrs { promote: true } } }
I20240501 20:02:10.670740 29540 catalog_manager.cc:5484] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130 reported cstate change: config changed from index -1 to 2, NON_VOTER 43f052a8add942cf9d91f97be38da636 (127.24.69.132) added. New cstate: current_term: 2 leader_uuid: "e93d3d097d994a1992554c82c6881b83" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } attrs { replace: true } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } } peers { permanent_uuid: "43f052a8add942cf9d91f97be38da636" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 38453 } attrs { promote: true } } }
I20240501 20:02:11.146926 30240 ts_tablet_manager.cc:902] T cd88640130a342d899a02a89c3a9fcfa P 43f052a8add942cf9d91f97be38da636: Initiating tablet copy from peer 5947681784a745eea2d6b0c97e234a4c (127.24.69.131:43209)
I20240501 20:02:11.150501 30240 tablet_copy_client.cc:308] T cd88640130a342d899a02a89c3a9fcfa P 43f052a8add942cf9d91f97be38da636: tablet copy: Beginning tablet copy session from remote peer at address 127.24.69.131:43209
I20240501 20:02:11.164465 30175 tablet_copy_service.cc:140] P 5947681784a745eea2d6b0c97e234a4c: Received BeginTabletCopySession request for tablet cd88640130a342d899a02a89c3a9fcfa from peer 43f052a8add942cf9d91f97be38da636 ({username='slave'} at 127.0.0.1:38994)
I20240501 20:02:11.164994 30175 tablet_copy_service.cc:161] P 5947681784a745eea2d6b0c97e234a4c: Beginning new tablet copy session on tablet cd88640130a342d899a02a89c3a9fcfa from peer 43f052a8add942cf9d91f97be38da636 at {username='slave'} at 127.0.0.1:38994: session id = 43f052a8add942cf9d91f97be38da636-cd88640130a342d899a02a89c3a9fcfa
I20240501 20:02:11.167979 30247 ts_tablet_manager.cc:902] T ed6ad07388d449cf8a8504a00f60dbd8 P 43f052a8add942cf9d91f97be38da636: Initiating tablet copy from peer e93d3d097d994a1992554c82c6881b83 (127.24.69.129:44161)
I20240501 20:02:11.173784 30175 tablet_copy_source_session.cc:205] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:02:11.177451 30240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd88640130a342d899a02a89c3a9fcfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:11.185045 30247 tablet_copy_client.cc:308] T ed6ad07388d449cf8a8504a00f60dbd8 P 43f052a8add942cf9d91f97be38da636: tablet copy: Beginning tablet copy session from remote peer at address 127.24.69.129:44161
I20240501 20:02:11.194257 30240 tablet_copy_client.cc:783] T cd88640130a342d899a02a89c3a9fcfa P 43f052a8add942cf9d91f97be38da636: tablet copy: Starting download of 0 data blocks...
I20240501 20:02:11.195255 30240 tablet_copy_client.cc:647] T cd88640130a342d899a02a89c3a9fcfa P 43f052a8add942cf9d91f97be38da636: tablet copy: Starting download of 1 WAL segments...
I20240501 20:02:11.197055 30253 ts_tablet_manager.cc:902] T 96a2762d729f41a3a34b013bf76d4702 P 43f052a8add942cf9d91f97be38da636: Initiating tablet copy from peer e93d3d097d994a1992554c82c6881b83 (127.24.69.129:44161)
I20240501 20:02:11.201282 30240 tablet_copy_client.cc:523] T cd88640130a342d899a02a89c3a9fcfa P 43f052a8add942cf9d91f97be38da636: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:02:11.207557 30014 tablet_copy_service.cc:140] P e93d3d097d994a1992554c82c6881b83: Received BeginTabletCopySession request for tablet ed6ad07388d449cf8a8504a00f60dbd8 from peer 43f052a8add942cf9d91f97be38da636 ({username='slave'} at 127.0.0.1:58200)
I20240501 20:02:11.208123 30014 tablet_copy_service.cc:161] P e93d3d097d994a1992554c82c6881b83: Beginning new tablet copy session on tablet ed6ad07388d449cf8a8504a00f60dbd8 from peer 43f052a8add942cf9d91f97be38da636 at {username='slave'} at 127.0.0.1:58200: session id = 43f052a8add942cf9d91f97be38da636-ed6ad07388d449cf8a8504a00f60dbd8
I20240501 20:02:11.213141 30240 tablet_bootstrap.cc:492] T cd88640130a342d899a02a89c3a9fcfa P 43f052a8add942cf9d91f97be38da636: Bootstrap starting.
I20240501 20:02:11.214907 30253 tablet_copy_client.cc:308] T 96a2762d729f41a3a34b013bf76d4702 P 43f052a8add942cf9d91f97be38da636: tablet copy: Beginning tablet copy session from remote peer at address 127.24.69.129:44161
I20240501 20:02:11.216795 30014 tablet_copy_source_session.cc:205] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:02:11.217437 30013 tablet_copy_service.cc:140] P e93d3d097d994a1992554c82c6881b83: Received BeginTabletCopySession request for tablet 96a2762d729f41a3a34b013bf76d4702 from peer 43f052a8add942cf9d91f97be38da636 ({username='slave'} at 127.0.0.1:58200)
I20240501 20:02:11.217980 30013 tablet_copy_service.cc:161] P e93d3d097d994a1992554c82c6881b83: Beginning new tablet copy session on tablet 96a2762d729f41a3a34b013bf76d4702 from peer 43f052a8add942cf9d91f97be38da636 at {username='slave'} at 127.0.0.1:58200: session id = 43f052a8add942cf9d91f97be38da636-96a2762d729f41a3a34b013bf76d4702
I20240501 20:02:11.220074 30247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed6ad07388d449cf8a8504a00f60dbd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:11.226213 30013 tablet_copy_source_session.cc:205] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:02:11.229612 30253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96a2762d729f41a3a34b013bf76d4702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:11.238804 30247 tablet_copy_client.cc:783] T ed6ad07388d449cf8a8504a00f60dbd8 P 43f052a8add942cf9d91f97be38da636: tablet copy: Starting download of 0 data blocks...
I20240501 20:02:11.239385 30247 tablet_copy_client.cc:647] T ed6ad07388d449cf8a8504a00f60dbd8 P 43f052a8add942cf9d91f97be38da636: tablet copy: Starting download of 1 WAL segments...
I20240501 20:02:11.242439 30240 tablet_bootstrap.cc:492] T cd88640130a342d899a02a89c3a9fcfa P 43f052a8add942cf9d91f97be38da636: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:02:11.242904 30247 tablet_copy_client.cc:523] T ed6ad07388d449cf8a8504a00f60dbd8 P 43f052a8add942cf9d91f97be38da636: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:02:11.243408 30240 tablet_bootstrap.cc:492] T cd88640130a342d899a02a89c3a9fcfa P 43f052a8add942cf9d91f97be38da636: Bootstrap complete.
I20240501 20:02:11.244191 30240 ts_tablet_manager.cc:1367] T cd88640130a342d899a02a89c3a9fcfa P 43f052a8add942cf9d91f97be38da636: Time spent bootstrapping tablet: real 0.031s user 0.013s sys 0.016s
I20240501 20:02:11.245232 30253 tablet_copy_client.cc:783] T 96a2762d729f41a3a34b013bf76d4702 P 43f052a8add942cf9d91f97be38da636: tablet copy: Starting download of 0 data blocks...
I20240501 20:02:11.245926 30253 tablet_copy_client.cc:647] T 96a2762d729f41a3a34b013bf76d4702 P 43f052a8add942cf9d91f97be38da636: tablet copy: Starting download of 1 WAL segments...
I20240501 20:02:11.247252 30240 raft_consensus.cc:348] T cd88640130a342d899a02a89c3a9fcfa P 43f052a8add942cf9d91f97be38da636 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } attrs { replace: true } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } } peers { permanent_uuid: "43f052a8add942cf9d91f97be38da636" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 38453 } attrs { promote: true } }
I20240501 20:02:11.248212 30240 raft_consensus.cc:729] T cd88640130a342d899a02a89c3a9fcfa P 43f052a8add942cf9d91f97be38da636 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 43f052a8add942cf9d91f97be38da636, State: Initialized, Role: LEARNER
I20240501 20:02:11.249483 30253 tablet_copy_client.cc:523] T 96a2762d729f41a3a34b013bf76d4702 P 43f052a8add942cf9d91f97be38da636: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:02:11.248946 30240 consensus_queue.cc:260] T cd88640130a342d899a02a89c3a9fcfa P 43f052a8add942cf9d91f97be38da636 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } attrs { replace: true } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } } peers { permanent_uuid: "43f052a8add942cf9d91f97be38da636" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 38453 } attrs { promote: true } }
I20240501 20:02:11.251456 30247 tablet_bootstrap.cc:492] T ed6ad07388d449cf8a8504a00f60dbd8 P 43f052a8add942cf9d91f97be38da636: Bootstrap starting.
I20240501 20:02:11.258092 30253 tablet_bootstrap.cc:492] T 96a2762d729f41a3a34b013bf76d4702 P 43f052a8add942cf9d91f97be38da636: Bootstrap starting.
I20240501 20:02:11.258858 30240 ts_tablet_manager.cc:1398] T cd88640130a342d899a02a89c3a9fcfa P 43f052a8add942cf9d91f97be38da636: Time spent starting tablet: real 0.014s user 0.005s sys 0.006s
I20240501 20:02:11.266610 30175 tablet_copy_service.cc:342] P 5947681784a745eea2d6b0c97e234a4c: Request end of tablet copy session 43f052a8add942cf9d91f97be38da636-cd88640130a342d899a02a89c3a9fcfa received from {username='slave'} at 127.0.0.1:38994
I20240501 20:02:11.267035 30175 tablet_copy_service.cc:433] P 5947681784a745eea2d6b0c97e234a4c: ending tablet copy session 43f052a8add942cf9d91f97be38da636-cd88640130a342d899a02a89c3a9fcfa on tablet cd88640130a342d899a02a89c3a9fcfa with peer 43f052a8add942cf9d91f97be38da636
I20240501 20:02:11.277463 30247 tablet_bootstrap.cc:492] T ed6ad07388d449cf8a8504a00f60dbd8 P 43f052a8add942cf9d91f97be38da636: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:02:11.278187 30247 tablet_bootstrap.cc:492] T ed6ad07388d449cf8a8504a00f60dbd8 P 43f052a8add942cf9d91f97be38da636: Bootstrap complete.
I20240501 20:02:11.278739 30247 ts_tablet_manager.cc:1367] T ed6ad07388d449cf8a8504a00f60dbd8 P 43f052a8add942cf9d91f97be38da636: Time spent bootstrapping tablet: real 0.028s user 0.023s sys 0.001s
I20240501 20:02:11.280278 30253 tablet_bootstrap.cc:492] T 96a2762d729f41a3a34b013bf76d4702 P 43f052a8add942cf9d91f97be38da636: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:02:11.281190 30253 tablet_bootstrap.cc:492] T 96a2762d729f41a3a34b013bf76d4702 P 43f052a8add942cf9d91f97be38da636: Bootstrap complete.
I20240501 20:02:11.280944 30247 raft_consensus.cc:348] T ed6ad07388d449cf8a8504a00f60dbd8 P 43f052a8add942cf9d91f97be38da636 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } attrs { replace: true } } peers { permanent_uuid: "43f052a8add942cf9d91f97be38da636" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 38453 } attrs { promote: true } }
I20240501 20:02:11.281839 30253 ts_tablet_manager.cc:1367] T 96a2762d729f41a3a34b013bf76d4702 P 43f052a8add942cf9d91f97be38da636: Time spent bootstrapping tablet: real 0.024s user 0.015s sys 0.008s
I20240501 20:02:11.281809 30247 raft_consensus.cc:729] T ed6ad07388d449cf8a8504a00f60dbd8 P 43f052a8add942cf9d91f97be38da636 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 43f052a8add942cf9d91f97be38da636, State: Initialized, Role: LEARNER
I20240501 20:02:11.282653 30247 consensus_queue.cc:260] T ed6ad07388d449cf8a8504a00f60dbd8 P 43f052a8add942cf9d91f97be38da636 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } attrs { replace: true } } peers { permanent_uuid: "43f052a8add942cf9d91f97be38da636" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 38453 } attrs { promote: true } }
I20240501 20:02:11.284745 30247 ts_tablet_manager.cc:1398] T ed6ad07388d449cf8a8504a00f60dbd8 P 43f052a8add942cf9d91f97be38da636: Time spent starting tablet: real 0.006s user 0.000s sys 0.008s
I20240501 20:02:11.285533 30253 raft_consensus.cc:348] T 96a2762d729f41a3a34b013bf76d4702 P 43f052a8add942cf9d91f97be38da636 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } attrs { replace: true } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } } peers { permanent_uuid: "43f052a8add942cf9d91f97be38da636" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 38453 } attrs { promote: true } }
I20240501 20:02:11.286453 30253 raft_consensus.cc:729] T 96a2762d729f41a3a34b013bf76d4702 P 43f052a8add942cf9d91f97be38da636 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 43f052a8add942cf9d91f97be38da636, State: Initialized, Role: LEARNER
I20240501 20:02:11.287078 30013 tablet_copy_service.cc:342] P e93d3d097d994a1992554c82c6881b83: Request end of tablet copy session 43f052a8add942cf9d91f97be38da636-ed6ad07388d449cf8a8504a00f60dbd8 received from {username='slave'} at 127.0.0.1:58200
I20240501 20:02:11.287618 30013 tablet_copy_service.cc:433] P e93d3d097d994a1992554c82c6881b83: ending tablet copy session 43f052a8add942cf9d91f97be38da636-ed6ad07388d449cf8a8504a00f60dbd8 on tablet ed6ad07388d449cf8a8504a00f60dbd8 with peer 43f052a8add942cf9d91f97be38da636
I20240501 20:02:11.287098 30253 consensus_queue.cc:260] T 96a2762d729f41a3a34b013bf76d4702 P 43f052a8add942cf9d91f97be38da636 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e93d3d097d994a1992554c82c6881b83" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 44161 } } peers { permanent_uuid: "1e7eecb723634a5da85a7c4d19108130" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 40933 } attrs { replace: true } } peers { permanent_uuid: "5947681784a745eea2d6b0c97e234a4c" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 43209 } } peers { permanent_uuid: "43f052a8add942cf9d91f97be38da636" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 38453 } attrs { promote: true } }
I20240501 20:02:11.290591 30253 ts_tablet_manager.cc:1398] T 96a2762d729f41a3a34b013bf76d4702 P 43f052a8add942cf9d91f97be38da636: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240501 20:02:11.292243 30013 tablet_copy_service.cc:342] P e93d3d097d994a1992554c82c6881b83: Request end of tablet copy session 43f052a8add942cf9d91f97be38da636-96a2762d729f41a3a34b013bf76d4702 received from {username='slave'} at 127.0.0.1:58200
I20240501 20:02:11.292622 30013 tablet_copy_service.cc:433] P e93d3d097d994a1992554c82c6881b83: ending tablet copy session 43f052a8add942cf9d91f97be38da636-96a2762d729f41a3a34b013bf76d4702 on tablet 96a2762d729f41a3a34b013bf76d4702 with peer 43f052a8add942cf9d91f97be38da636
I20240501 20:02:11.502220 24854 tablet_server.cc:178] TabletServer@127.24.69.129:44161 shutting down...
W20240501 20:02:11.508199 29585 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 8 similar messages]
W20240501 20:02:11.511301 29588 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 20:02:11.511781 29588 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 20:02:11.520800 29588 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 20:02:11.521167 29588 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240501 20:02:11.527050 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:02:11.527797 24854 tablet_replica.cc:330] stopping tablet replica
I20240501 20:02:11.528366 24854 raft_consensus.cc:2229] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:11.528820 24854 raft_consensus.cc:2258] T cd88640130a342d899a02a89c3a9fcfa P e93d3d097d994a1992554c82c6881b83 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:11.531147 24854 tablet_replica.cc:330] stopping tablet replica
I20240501 20:02:11.531803 24854 raft_consensus.cc:2229] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [term 2 LEADER]: Raft consensus shutting down.
I20240501 20:02:11.532734 24854 raft_consensus.cc:2258] T 7b3d39a2ed344577b6712279908a54ff P e93d3d097d994a1992554c82c6881b83 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:11.534607 24854 tablet_replica.cc:330] stopping tablet replica
I20240501 20:02:11.535302 24854 raft_consensus.cc:2229] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [term 2 LEADER]: Raft consensus shutting down.
I20240501 20:02:11.536157 24854 raft_consensus.cc:2258] T 96a2762d729f41a3a34b013bf76d4702 P e93d3d097d994a1992554c82c6881b83 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:11.538569 24854 tablet_replica.cc:330] stopping tablet replica
I20240501 20:02:11.539111 24854 raft_consensus.cc:2229] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 3 LEADER]: Raft consensus shutting down.
I20240501 20:02:11.540002 24854 raft_consensus.cc:2258] T ed6ad07388d449cf8a8504a00f60dbd8 P e93d3d097d994a1992554c82c6881b83 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:11.565114 24854 tablet_server.cc:194] TabletServer@127.24.69.129:44161 shutdown complete.
I20240501 20:02:11.577349 24854 tablet_server.cc:178] TabletServer@127.24.69.130:40933 shutting down...
I20240501 20:02:11.597165 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:02:11.597719 24854 tablet_replica.cc:330] stopping tablet replica
I20240501 20:02:11.598359 24854 raft_consensus.cc:2229] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:11.598809 24854 raft_consensus.cc:2258] T cd88640130a342d899a02a89c3a9fcfa P 1e7eecb723634a5da85a7c4d19108130 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:11.601061 24854 tablet_replica.cc:330] stopping tablet replica
I20240501 20:02:11.601601 24854 raft_consensus.cc:2229] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:11.602142 24854 raft_consensus.cc:2258] T 7b3d39a2ed344577b6712279908a54ff P 1e7eecb723634a5da85a7c4d19108130 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:11.603823 24854 tablet_replica.cc:330] stopping tablet replica
I20240501 20:02:11.604379 24854 raft_consensus.cc:2229] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:11.604820 24854 raft_consensus.cc:2258] T 96a2762d729f41a3a34b013bf76d4702 P 1e7eecb723634a5da85a7c4d19108130 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:11.606832 24854 tablet_replica.cc:330] stopping tablet replica
I20240501 20:02:11.607273 24854 raft_consensus.cc:2229] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:11.607686 24854 raft_consensus.cc:2258] T ed6ad07388d449cf8a8504a00f60dbd8 P 1e7eecb723634a5da85a7c4d19108130 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:11.629050 24854 tablet_server.cc:194] TabletServer@127.24.69.130:40933 shutdown complete.
I20240501 20:02:11.640690 24854 tablet_server.cc:178] TabletServer@127.24.69.131:43209 shutting down...
W20240501 20:02:11.644472 29588 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 20:02:11.645632 29588 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240501 20:02:11.658464 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:02:11.659039 24854 tablet_replica.cc:330] stopping tablet replica
I20240501 20:02:11.659543 24854 raft_consensus.cc:2229] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 3 LEADER]: Raft consensus shutting down.
I20240501 20:02:11.660488 24854 raft_consensus.cc:2258] T cd88640130a342d899a02a89c3a9fcfa P 5947681784a745eea2d6b0c97e234a4c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:11.662330 24854 tablet_replica.cc:330] stopping tablet replica
I20240501 20:02:11.662832 24854 raft_consensus.cc:2229] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:11.663245 24854 raft_consensus.cc:2258] T 7b3d39a2ed344577b6712279908a54ff P 5947681784a745eea2d6b0c97e234a4c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:11.664873 24854 tablet_replica.cc:330] stopping tablet replica
I20240501 20:02:11.665308 24854 raft_consensus.cc:2229] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:11.665700 24854 raft_consensus.cc:2258] T 96a2762d729f41a3a34b013bf76d4702 P 5947681784a745eea2d6b0c97e234a4c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:11.667542 24854 tablet_replica.cc:330] stopping tablet replica
I20240501 20:02:11.667968 24854 raft_consensus.cc:2229] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:11.668450 24854 raft_consensus.cc:2258] T ed6ad07388d449cf8a8504a00f60dbd8 P 5947681784a745eea2d6b0c97e234a4c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:11.688978 24854 tablet_server.cc:194] TabletServer@127.24.69.131:43209 shutdown complete.
I20240501 20:02:11.700235 24854 tablet_server.cc:178] TabletServer@127.24.69.132:0 shutting down...
I20240501 20:02:11.723764 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:02:11.724916 24854 tablet_replica.cc:330] T 96a2762d729f41a3a34b013bf76d4702 P 43f052a8add942cf9d91f97be38da636: stopping tablet replica
I20240501 20:02:11.725518 24854 raft_consensus.cc:2229] T 96a2762d729f41a3a34b013bf76d4702 P 43f052a8add942cf9d91f97be38da636 [term 2 LEARNER]: Raft consensus shutting down.
I20240501 20:02:11.725914 24854 raft_consensus.cc:2258] T 96a2762d729f41a3a34b013bf76d4702 P 43f052a8add942cf9d91f97be38da636 [term 2 LEARNER]: Raft consensus is shut down!
I20240501 20:02:11.727712 24854 tablet_replica.cc:330] T ed6ad07388d449cf8a8504a00f60dbd8 P 43f052a8add942cf9d91f97be38da636: stopping tablet replica
I20240501 20:02:11.728184 24854 raft_consensus.cc:2229] T ed6ad07388d449cf8a8504a00f60dbd8 P 43f052a8add942cf9d91f97be38da636 [term 3 LEARNER]: Raft consensus shutting down.
I20240501 20:02:11.728583 24854 raft_consensus.cc:2258] T ed6ad07388d449cf8a8504a00f60dbd8 P 43f052a8add942cf9d91f97be38da636 [term 3 LEARNER]: Raft consensus is shut down!
I20240501 20:02:11.730198 24854 tablet_replica.cc:330] T cd88640130a342d899a02a89c3a9fcfa P 43f052a8add942cf9d91f97be38da636: stopping tablet replica
I20240501 20:02:11.730621 24854 raft_consensus.cc:2229] T cd88640130a342d899a02a89c3a9fcfa P 43f052a8add942cf9d91f97be38da636 [term 3 LEARNER]: Raft consensus shutting down.
I20240501 20:02:11.730942 24854 raft_consensus.cc:2258] T cd88640130a342d899a02a89c3a9fcfa P 43f052a8add942cf9d91f97be38da636 [term 3 LEARNER]: Raft consensus is shut down!
I20240501 20:02:11.751477 24854 tablet_server.cc:194] TabletServer@127.24.69.132:0 shutdown complete.
I20240501 20:02:11.763703 24854 master.cc:537] Master@127.24.69.190:35105 shutting down...
I20240501 20:02:11.779958 24854 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3c19b51a8fbb4d1388b0a38650928c4d [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:02:11.780504 24854 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3c19b51a8fbb4d1388b0a38650928c4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:11.780865 24854 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3c19b51a8fbb4d1388b0a38650928c4d: stopping tablet replica
I20240501 20:02:11.798400 24854 master.cc:552] Master@127.24.69.190:35105 shutdown complete.
[ OK ] AutoRebalancerTest.TestHandlingFailedTservers (12614 ms)
[ RUN ] AutoRebalancerTest.TestDeletedTables
I20240501 20:02:11.835343 24854 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.69.190:40685
I20240501 20:02:11.836390 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:02:11.841611 30263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:11.842171 30264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:11.842916 30266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:02:11.843425 24854 server_base.cc:1034] running on GCE node
I20240501 20:02:11.844151 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:02:11.844367 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:02:11.844516 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593731844498 us; error 0 us; skew 500 ppm
I20240501 20:02:11.845003 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:02:11.847128 24854 webserver.cc:460] Webserver started at http://127.24.69.190:41033/ using document root <none> and password file <none>
I20240501 20:02:11.847569 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:02:11.847734 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:02:11.847971 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:02:11.849056 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/master-0-root/instance:
uuid: "1191fc0bfcfd4bb9b091152625e0abb4"
format_stamp: "Formatted at 2024-05-01 20:02:11 on dist-test-slave-z8x3"
I20240501 20:02:11.853261 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20240501 20:02:11.856093 30271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:11.856806 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240501 20:02:11.857054 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/master-0-root
uuid: "1191fc0bfcfd4bb9b091152625e0abb4"
format_stamp: "Formatted at 2024-05-01 20:02:11 on dist-test-slave-z8x3"
I20240501 20:02:11.857319 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:02:11.870041 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:02:11.871053 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:02:11.905671 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.190:40685
I20240501 20:02:11.905766 30322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.190:40685 every 8 connection(s)
I20240501 20:02:11.909250 30323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:11.918789 30323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1191fc0bfcfd4bb9b091152625e0abb4: Bootstrap starting.
I20240501 20:02:11.922719 30323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1191fc0bfcfd4bb9b091152625e0abb4: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:11.926620 30323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1191fc0bfcfd4bb9b091152625e0abb4: No bootstrap required, opened a new log
I20240501 20:02:11.928543 30323 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1191fc0bfcfd4bb9b091152625e0abb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1191fc0bfcfd4bb9b091152625e0abb4" member_type: VOTER }
I20240501 20:02:11.928913 30323 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1191fc0bfcfd4bb9b091152625e0abb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:11.929145 30323 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1191fc0bfcfd4bb9b091152625e0abb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1191fc0bfcfd4bb9b091152625e0abb4, State: Initialized, Role: FOLLOWER
I20240501 20:02:11.929694 30323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1191fc0bfcfd4bb9b091152625e0abb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1191fc0bfcfd4bb9b091152625e0abb4" member_type: VOTER }
I20240501 20:02:11.930179 30323 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1191fc0bfcfd4bb9b091152625e0abb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:02:11.930377 30323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1191fc0bfcfd4bb9b091152625e0abb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:02:11.930595 30323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1191fc0bfcfd4bb9b091152625e0abb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:11.934830 30323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1191fc0bfcfd4bb9b091152625e0abb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1191fc0bfcfd4bb9b091152625e0abb4" member_type: VOTER }
I20240501 20:02:11.935309 30323 leader_election.cc:302] T 00000000000000000000000000000000 P 1191fc0bfcfd4bb9b091152625e0abb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1191fc0bfcfd4bb9b091152625e0abb4; no voters:
I20240501 20:02:11.936424 30323 leader_election.cc:288] T 00000000000000000000000000000000 P 1191fc0bfcfd4bb9b091152625e0abb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:02:11.936803 30326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1191fc0bfcfd4bb9b091152625e0abb4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:02:11.938091 30326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1191fc0bfcfd4bb9b091152625e0abb4 [term 1 LEADER]: Becoming Leader. State: Replica: 1191fc0bfcfd4bb9b091152625e0abb4, State: Running, Role: LEADER
I20240501 20:02:11.938632 30326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1191fc0bfcfd4bb9b091152625e0abb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1191fc0bfcfd4bb9b091152625e0abb4" member_type: VOTER }
I20240501 20:02:11.939152 30323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1191fc0bfcfd4bb9b091152625e0abb4 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:02:11.941525 30327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1191fc0bfcfd4bb9b091152625e0abb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1191fc0bfcfd4bb9b091152625e0abb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1191fc0bfcfd4bb9b091152625e0abb4" member_type: VOTER } }
I20240501 20:02:11.941555 30328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1191fc0bfcfd4bb9b091152625e0abb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1191fc0bfcfd4bb9b091152625e0abb4. Latest consensus state: current_term: 1 leader_uuid: "1191fc0bfcfd4bb9b091152625e0abb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1191fc0bfcfd4bb9b091152625e0abb4" member_type: VOTER } }
I20240501 20:02:11.942335 30328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1191fc0bfcfd4bb9b091152625e0abb4 [sys.catalog]: This master's current role is: LEADER
I20240501 20:02:11.942330 30327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1191fc0bfcfd4bb9b091152625e0abb4 [sys.catalog]: This master's current role is: LEADER
I20240501 20:02:11.945845 30331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:02:11.950816 30331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:02:11.953707 24854 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 20:02:11.958850 30331 catalog_manager.cc:1347] Generated new cluster ID: 3a27f7623671483c86f32b49f24238d7
I20240501 20:02:11.959158 30331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:02:11.991683 30331 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:02:11.992946 30331 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:02:12.009734 30331 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1191fc0bfcfd4bb9b091152625e0abb4: Generated new TSK 0
I20240501 20:02:12.010283 30331 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:02:12.020361 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:02:12.026511 30344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:12.027194 30345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:12.028388 30347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:02:12.029094 24854 server_base.cc:1034] running on GCE node
I20240501 20:02:12.030519 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:02:12.030712 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:02:12.030833 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593732030821 us; error 0 us; skew 500 ppm
I20240501 20:02:12.031292 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:02:12.033516 24854 webserver.cc:460] Webserver started at http://127.24.69.129:41731/ using document root <none> and password file <none>
I20240501 20:02:12.033941 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:02:12.034096 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:02:12.034296 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:02:12.035295 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-0-root/instance:
uuid: "81885e2958c34da5ae5fe78b749d6e99"
format_stamp: "Formatted at 2024-05-01 20:02:12 on dist-test-slave-z8x3"
I20240501 20:02:12.039543 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20240501 20:02:12.042881 30352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:12.043668 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:02:12.043947 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-0-root
uuid: "81885e2958c34da5ae5fe78b749d6e99"
format_stamp: "Formatted at 2024-05-01 20:02:12 on dist-test-slave-z8x3"
I20240501 20:02:12.044224 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:02:12.079012 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:02:12.080070 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:02:12.081512 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:02:12.083670 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:02:12.083868 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:12.084096 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:02:12.084249 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:12.123992 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.129:35925
I20240501 20:02:12.124050 30414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.129:35925 every 8 connection(s)
I20240501 20:02:12.128656 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:02:12.136189 30419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:12.137383 30420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:12.141819 30422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:02:12.142835 24854 server_base.cc:1034] running on GCE node
I20240501 20:02:12.143051 30415 heartbeater.cc:347] Connected to a master server at 127.24.69.190:40685
I20240501 20:02:12.143460 30415 heartbeater.cc:464] Registering TS with master...
I20240501 20:02:12.143988 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:02:12.144290 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:02:12.144373 30415 heartbeater.cc:511] Master 127.24.69.190:40685 requested a full tablet report, sending...
I20240501 20:02:12.144526 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593732144506 us; error 0 us; skew 500 ppm
I20240501 20:02:12.145308 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:02:12.146718 30288 ts_manager.cc:194] Registered new tserver with Master: 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925)
I20240501 20:02:12.148087 24854 webserver.cc:460] Webserver started at http://127.24.69.130:33747/ using document root <none> and password file <none>
I20240501 20:02:12.148741 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:02:12.148967 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:02:12.149165 30288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41464
I20240501 20:02:12.149272 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:02:12.150772 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-1-root/instance:
uuid: "e4b430a4ca81472ba0d5c42a4fa6f604"
format_stamp: "Formatted at 2024-05-01 20:02:12 on dist-test-slave-z8x3"
I20240501 20:02:12.155115 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 20:02:12.158087 30427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:12.158991 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:02:12.159243 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-1-root
uuid: "e4b430a4ca81472ba0d5c42a4fa6f604"
format_stamp: "Formatted at 2024-05-01 20:02:12 on dist-test-slave-z8x3"
I20240501 20:02:12.159487 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:02:12.172446 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:02:12.173385 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:02:12.174741 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:02:12.176973 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:02:12.177176 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:12.177397 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:02:12.177546 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:12.213490 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.130:43685
I20240501 20:02:12.213598 30489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.130:43685 every 8 connection(s)
I20240501 20:02:12.218123 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:02:12.224951 30493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:12.227510 30494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:02:12.229784 24854 server_base.cc:1034] running on GCE node
I20240501 20:02:12.230093 30490 heartbeater.cc:347] Connected to a master server at 127.24.69.190:40685
I20240501 20:02:12.230572 30490 heartbeater.cc:464] Registering TS with master...
W20240501 20:02:12.230659 30496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:02:12.231503 30490 heartbeater.cc:511] Master 127.24.69.190:40685 requested a full tablet report, sending...
I20240501 20:02:12.231590 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:02:12.231905 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:02:12.232069 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593732232050 us; error 0 us; skew 500 ppm
I20240501 20:02:12.232662 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:02:12.233357 30288 ts_manager.cc:194] Registered new tserver with Master: e4b430a4ca81472ba0d5c42a4fa6f604 (127.24.69.130:43685)
I20240501 20:02:12.234777 30288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41470
I20240501 20:02:12.235193 24854 webserver.cc:460] Webserver started at http://127.24.69.131:33427/ using document root <none> and password file <none>
I20240501 20:02:12.235716 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:02:12.235958 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:02:12.236207 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:02:12.237283 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-2-root/instance:
uuid: "747b41f97e7246cd8a42097546f44520"
format_stamp: "Formatted at 2024-05-01 20:02:12 on dist-test-slave-z8x3"
I20240501 20:02:12.241528 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240501 20:02:12.244590 30501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:12.245360 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 20:02:12.245625 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-2-root
uuid: "747b41f97e7246cd8a42097546f44520"
format_stamp: "Formatted at 2024-05-01 20:02:12 on dist-test-slave-z8x3"
I20240501 20:02:12.245894 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:02:12.265543 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:02:12.266549 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:02:12.267922 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:02:12.270069 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:02:12.270275 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:12.270498 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:02:12.270648 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:12.306730 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.131:37749
I20240501 20:02:12.306824 30563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.131:37749 every 8 connection(s)
I20240501 20:02:12.319414 30564 heartbeater.cc:347] Connected to a master server at 127.24.69.190:40685
I20240501 20:02:12.319749 30564 heartbeater.cc:464] Registering TS with master...
I20240501 20:02:12.320430 30564 heartbeater.cc:511] Master 127.24.69.190:40685 requested a full tablet report, sending...
I20240501 20:02:12.322202 30288 ts_manager.cc:194] Registered new tserver with Master: 747b41f97e7246cd8a42097546f44520 (127.24.69.131:37749)
I20240501 20:02:12.323105 24854 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013333384s
I20240501 20:02:12.323515 30288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41482
I20240501 20:02:13.151955 30415 heartbeater.cc:503] Master 127.24.69.190:40685 was elected leader, sending a full tablet report...
I20240501 20:02:13.237043 30490 heartbeater.cc:503] Master 127.24.69.190:40685 was elected leader, sending a full tablet report...
I20240501 20:02:13.325956 30564 heartbeater.cc:503] Master 127.24.69.190:40685 was elected leader, sending a full tablet report...
I20240501 20:02:13.354797 24854 test_util.cc:268] Using random seed: -160881385
I20240501 20:02:13.381773 30288 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41498:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 20:02:13.384058 30288 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:02:13.441610 30380 tablet_service.cc:1467] Processing CreateTablet for tablet 3378dd5a0ac64b3e852cb61ee61cec8b (DEFAULT_TABLE table=test-workload [id=331b4d80f40e4828a7954cbafaf2e5e2]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240501 20:02:13.442494 30379 tablet_service.cc:1467] Processing CreateTablet for tablet d3fb659b47994126bf3c60689f0b13b5 (DEFAULT_TABLE table=test-workload [id=331b4d80f40e4828a7954cbafaf2e5e2]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240501 20:02:13.442999 30380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3378dd5a0ac64b3e852cb61ee61cec8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:13.443841 30379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3fb659b47994126bf3c60689f0b13b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:13.444098 30377 tablet_service.cc:1467] Processing CreateTablet for tablet dfd8f9eb29d745589b4aa63188a0c2aa (DEFAULT_TABLE table=test-workload [id=331b4d80f40e4828a7954cbafaf2e5e2]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240501 20:02:13.445397 30377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfd8f9eb29d745589b4aa63188a0c2aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:13.446686 30529 tablet_service.cc:1467] Processing CreateTablet for tablet 3378dd5a0ac64b3e852cb61ee61cec8b (DEFAULT_TABLE table=test-workload [id=331b4d80f40e4828a7954cbafaf2e5e2]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240501 20:02:13.447266 30378 tablet_service.cc:1467] Processing CreateTablet for tablet 6d8bde677f5e4f87802eda8373720358 (DEFAULT_TABLE table=test-workload [id=331b4d80f40e4828a7954cbafaf2e5e2]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240501 20:02:13.447969 30529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3378dd5a0ac64b3e852cb61ee61cec8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:13.448527 30378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d8bde677f5e4f87802eda8373720358. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:13.449290 30528 tablet_service.cc:1467] Processing CreateTablet for tablet d3fb659b47994126bf3c60689f0b13b5 (DEFAULT_TABLE table=test-workload [id=331b4d80f40e4828a7954cbafaf2e5e2]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240501 20:02:13.451299 30527 tablet_service.cc:1467] Processing CreateTablet for tablet 6d8bde677f5e4f87802eda8373720358 (DEFAULT_TABLE table=test-workload [id=331b4d80f40e4828a7954cbafaf2e5e2]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240501 20:02:13.453342 30526 tablet_service.cc:1467] Processing CreateTablet for tablet dfd8f9eb29d745589b4aa63188a0c2aa (DEFAULT_TABLE table=test-workload [id=331b4d80f40e4828a7954cbafaf2e5e2]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240501 20:02:13.457722 30526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfd8f9eb29d745589b4aa63188a0c2aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:13.460839 30528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3fb659b47994126bf3c60689f0b13b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:13.464354 30527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d8bde677f5e4f87802eda8373720358. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:13.468087 30584 tablet_bootstrap.cc:492] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap starting.
I20240501 20:02:13.475194 30584 tablet_bootstrap.cc:654] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:13.486824 30585 tablet_bootstrap.cc:492] T d3fb659b47994126bf3c60689f0b13b5 P 747b41f97e7246cd8a42097546f44520: Bootstrap starting.
I20240501 20:02:13.492754 30585 tablet_bootstrap.cc:654] T d3fb659b47994126bf3c60689f0b13b5 P 747b41f97e7246cd8a42097546f44520: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:13.494179 30584 tablet_bootstrap.cc:492] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99: No bootstrap required, opened a new log
I20240501 20:02:13.494732 30584 ts_tablet_manager.cc:1367] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99: Time spent bootstrapping tablet: real 0.027s user 0.015s sys 0.004s
I20240501 20:02:13.495513 30453 tablet_service.cc:1467] Processing CreateTablet for tablet 6d8bde677f5e4f87802eda8373720358 (DEFAULT_TABLE table=test-workload [id=331b4d80f40e4828a7954cbafaf2e5e2]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240501 20:02:13.496802 30453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d8bde677f5e4f87802eda8373720358. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:13.496789 30452 tablet_service.cc:1467] Processing CreateTablet for tablet dfd8f9eb29d745589b4aa63188a0c2aa (DEFAULT_TABLE table=test-workload [id=331b4d80f40e4828a7954cbafaf2e5e2]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240501 20:02:13.497872 30584 raft_consensus.cc:348] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } }
I20240501 20:02:13.498879 30584 raft_consensus.cc:374] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:13.498739 30454 tablet_service.cc:1467] Processing CreateTablet for tablet d3fb659b47994126bf3c60689f0b13b5 (DEFAULT_TABLE table=test-workload [id=331b4d80f40e4828a7954cbafaf2e5e2]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240501 20:02:13.499287 30584 raft_consensus.cc:729] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81885e2958c34da5ae5fe78b749d6e99, State: Initialized, Role: FOLLOWER
I20240501 20:02:13.500103 30454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3fb659b47994126bf3c60689f0b13b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:13.500217 30584 consensus_queue.cc:260] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } }
I20240501 20:02:13.504765 30585 tablet_bootstrap.cc:492] T d3fb659b47994126bf3c60689f0b13b5 P 747b41f97e7246cd8a42097546f44520: No bootstrap required, opened a new log
I20240501 20:02:13.505537 30585 ts_tablet_manager.cc:1367] T d3fb659b47994126bf3c60689f0b13b5 P 747b41f97e7246cd8a42097546f44520: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.012s
I20240501 20:02:13.505334 30455 tablet_service.cc:1467] Processing CreateTablet for tablet 3378dd5a0ac64b3e852cb61ee61cec8b (DEFAULT_TABLE table=test-workload [id=331b4d80f40e4828a7954cbafaf2e5e2]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240501 20:02:13.506234 30452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfd8f9eb29d745589b4aa63188a0c2aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:13.507076 30455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3378dd5a0ac64b3e852cb61ee61cec8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:13.519932 30584 ts_tablet_manager.cc:1398] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99: Time spent starting tablet: real 0.025s user 0.008s sys 0.016s
I20240501 20:02:13.519304 30585 raft_consensus.cc:348] T d3fb659b47994126bf3c60689f0b13b5 P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.521330 30585 raft_consensus.cc:374] T d3fb659b47994126bf3c60689f0b13b5 P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:13.521673 30585 raft_consensus.cc:729] T d3fb659b47994126bf3c60689f0b13b5 P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 747b41f97e7246cd8a42097546f44520, State: Initialized, Role: FOLLOWER
I20240501 20:02:13.522416 30585 consensus_queue.cc:260] T d3fb659b47994126bf3c60689f0b13b5 P 747b41f97e7246cd8a42097546f44520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.521059 30584 tablet_bootstrap.cc:492] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap starting.
I20240501 20:02:13.529357 30584 tablet_bootstrap.cc:654] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:13.538242 30589 tablet_bootstrap.cc:492] T 6d8bde677f5e4f87802eda8373720358 P e4b430a4ca81472ba0d5c42a4fa6f604: Bootstrap starting.
I20240501 20:02:13.541050 30585 ts_tablet_manager.cc:1398] T d3fb659b47994126bf3c60689f0b13b5 P 747b41f97e7246cd8a42097546f44520: Time spent starting tablet: real 0.025s user 0.007s sys 0.009s
I20240501 20:02:13.542443 30585 tablet_bootstrap.cc:492] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520: Bootstrap starting.
I20240501 20:02:13.548347 30589 tablet_bootstrap.cc:654] T 6d8bde677f5e4f87802eda8373720358 P e4b430a4ca81472ba0d5c42a4fa6f604: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:13.549121 30584 tablet_bootstrap.cc:492] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99: No bootstrap required, opened a new log
I20240501 20:02:13.549710 30584 ts_tablet_manager.cc:1367] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.014s
I20240501 20:02:13.550288 30585 tablet_bootstrap.cc:654] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:13.552747 30584 raft_consensus.cc:348] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.553869 30584 raft_consensus.cc:374] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:13.554306 30584 raft_consensus.cc:729] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81885e2958c34da5ae5fe78b749d6e99, State: Initialized, Role: FOLLOWER
I20240501 20:02:13.555442 30584 consensus_queue.cc:260] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.557242 30588 raft_consensus.cc:482] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:13.557694 30588 raft_consensus.cc:504] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.559253 30589 tablet_bootstrap.cc:492] T 6d8bde677f5e4f87802eda8373720358 P e4b430a4ca81472ba0d5c42a4fa6f604: No bootstrap required, opened a new log
I20240501 20:02:13.559751 30585 tablet_bootstrap.cc:492] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520: No bootstrap required, opened a new log
I20240501 20:02:13.559890 30589 ts_tablet_manager.cc:1367] T 6d8bde677f5e4f87802eda8373720358 P e4b430a4ca81472ba0d5c42a4fa6f604: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.003s
I20240501 20:02:13.560506 30585 ts_tablet_manager.cc:1367] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.006s
I20240501 20:02:13.562971 30589 raft_consensus.cc:348] T 6d8bde677f5e4f87802eda8373720358 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.563753 30589 raft_consensus.cc:374] T 6d8bde677f5e4f87802eda8373720358 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:13.564065 30589 raft_consensus.cc:729] T 6d8bde677f5e4f87802eda8373720358 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b430a4ca81472ba0d5c42a4fa6f604, State: Initialized, Role: FOLLOWER
I20240501 20:02:13.563858 30585 raft_consensus.cc:348] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.565133 30585 raft_consensus.cc:374] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:13.565047 30589 consensus_queue.cc:260] T 6d8bde677f5e4f87802eda8373720358 P e4b430a4ca81472ba0d5c42a4fa6f604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.565608 30585 raft_consensus.cc:729] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 747b41f97e7246cd8a42097546f44520, State: Initialized, Role: FOLLOWER
I20240501 20:02:13.560143 30588 leader_election.cc:288] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4b430a4ca81472ba0d5c42a4fa6f604 (127.24.69.130:43685), 747b41f97e7246cd8a42097546f44520 (127.24.69.131:37749)
I20240501 20:02:13.571491 30585 consensus_queue.cc:260] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.579281 30585 ts_tablet_manager.cc:1398] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520: Time spent starting tablet: real 0.018s user 0.008s sys 0.000s
I20240501 20:02:13.580698 30585 tablet_bootstrap.cc:492] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520: Bootstrap starting.
I20240501 20:02:13.593977 30584 ts_tablet_manager.cc:1398] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99: Time spent starting tablet: real 0.044s user 0.020s sys 0.021s
I20240501 20:02:13.595803 30589 ts_tablet_manager.cc:1398] T 6d8bde677f5e4f87802eda8373720358 P e4b430a4ca81472ba0d5c42a4fa6f604: Time spent starting tablet: real 0.035s user 0.013s sys 0.020s
I20240501 20:02:13.597770 30584 tablet_bootstrap.cc:492] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap starting.
I20240501 20:02:13.598690 30589 tablet_bootstrap.cc:492] T 3378dd5a0ac64b3e852cb61ee61cec8b P e4b430a4ca81472ba0d5c42a4fa6f604: Bootstrap starting.
I20240501 20:02:13.598994 30465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8bde677f5e4f87802eda8373720358" candidate_uuid: "81885e2958c34da5ae5fe78b749d6e99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" is_pre_election: true
I20240501 20:02:13.599733 30465 raft_consensus.cc:2454] T 6d8bde677f5e4f87802eda8373720358 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81885e2958c34da5ae5fe78b749d6e99 in term 0.
I20240501 20:02:13.601085 30353 leader_election.cc:302] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81885e2958c34da5ae5fe78b749d6e99, e4b430a4ca81472ba0d5c42a4fa6f604; no voters:
I20240501 20:02:13.602546 30584 tablet_bootstrap.cc:654] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:13.604298 30589 tablet_bootstrap.cc:654] T 3378dd5a0ac64b3e852cb61ee61cec8b P e4b430a4ca81472ba0d5c42a4fa6f604: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:13.605928 30594 raft_consensus.cc:2781] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:02:13.596653 30585 tablet_bootstrap.cc:654] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:13.606545 30594 raft_consensus.cc:482] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:02:13.607167 30594 raft_consensus.cc:3037] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:13.612313 30539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8bde677f5e4f87802eda8373720358" candidate_uuid: "81885e2958c34da5ae5fe78b749d6e99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "747b41f97e7246cd8a42097546f44520" is_pre_election: true
I20240501 20:02:13.613216 30539 raft_consensus.cc:2454] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81885e2958c34da5ae5fe78b749d6e99 in term 0.
I20240501 20:02:13.623653 30589 tablet_bootstrap.cc:492] T 3378dd5a0ac64b3e852cb61ee61cec8b P e4b430a4ca81472ba0d5c42a4fa6f604: No bootstrap required, opened a new log
I20240501 20:02:13.624169 30589 ts_tablet_manager.cc:1367] T 3378dd5a0ac64b3e852cb61ee61cec8b P e4b430a4ca81472ba0d5c42a4fa6f604: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.006s
I20240501 20:02:13.624747 30585 tablet_bootstrap.cc:492] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520: No bootstrap required, opened a new log
I20240501 20:02:13.625223 30585 ts_tablet_manager.cc:1367] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520: Time spent bootstrapping tablet: real 0.045s user 0.015s sys 0.002s
I20240501 20:02:13.626744 30589 raft_consensus.cc:348] T 3378dd5a0ac64b3e852cb61ee61cec8b P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } }
I20240501 20:02:13.627499 30589 raft_consensus.cc:374] T 3378dd5a0ac64b3e852cb61ee61cec8b P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:13.627911 30589 raft_consensus.cc:729] T 3378dd5a0ac64b3e852cb61ee61cec8b P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b430a4ca81472ba0d5c42a4fa6f604, State: Initialized, Role: FOLLOWER
I20240501 20:02:13.628775 30594 raft_consensus.cc:504] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.628841 30589 consensus_queue.cc:260] T 3378dd5a0ac64b3e852cb61ee61cec8b P e4b430a4ca81472ba0d5c42a4fa6f604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } }
I20240501 20:02:13.627573 30585 raft_consensus.cc:348] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.630231 30585 raft_consensus.cc:374] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:13.630347 30584 tablet_bootstrap.cc:492] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99: No bootstrap required, opened a new log
I20240501 20:02:13.630586 30585 raft_consensus.cc:729] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 747b41f97e7246cd8a42097546f44520, State: Initialized, Role: FOLLOWER
I20240501 20:02:13.630863 30584 ts_tablet_manager.cc:1367] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99: Time spent bootstrapping tablet: real 0.033s user 0.027s sys 0.005s
I20240501 20:02:13.631212 30589 ts_tablet_manager.cc:1398] T 3378dd5a0ac64b3e852cb61ee61cec8b P e4b430a4ca81472ba0d5c42a4fa6f604: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240501 20:02:13.631485 30585 consensus_queue.cc:260] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.632144 30589 tablet_bootstrap.cc:492] T dfd8f9eb29d745589b4aa63188a0c2aa P e4b430a4ca81472ba0d5c42a4fa6f604: Bootstrap starting.
I20240501 20:02:13.633502 30584 raft_consensus.cc:348] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.634281 30584 raft_consensus.cc:374] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:13.634601 30584 raft_consensus.cc:729] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81885e2958c34da5ae5fe78b749d6e99, State: Initialized, Role: FOLLOWER
I20240501 20:02:13.635404 30584 consensus_queue.cc:260] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.637652 30584 ts_tablet_manager.cc:1398] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 20:02:13.637835 30589 tablet_bootstrap.cc:654] T dfd8f9eb29d745589b4aa63188a0c2aa P e4b430a4ca81472ba0d5c42a4fa6f604: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:13.640316 30585 ts_tablet_manager.cc:1398] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20240501 20:02:13.640524 30584 tablet_bootstrap.cc:492] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap starting.
I20240501 20:02:13.646656 30584 tablet_bootstrap.cc:654] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:13.648872 30465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8bde677f5e4f87802eda8373720358" candidate_uuid: "81885e2958c34da5ae5fe78b749d6e99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604"
I20240501 20:02:13.649581 30465 raft_consensus.cc:3037] T 6d8bde677f5e4f87802eda8373720358 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:13.651382 30589 tablet_bootstrap.cc:492] T dfd8f9eb29d745589b4aa63188a0c2aa P e4b430a4ca81472ba0d5c42a4fa6f604: No bootstrap required, opened a new log
I20240501 20:02:13.651846 30589 ts_tablet_manager.cc:1367] T dfd8f9eb29d745589b4aa63188a0c2aa P e4b430a4ca81472ba0d5c42a4fa6f604: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.004s
I20240501 20:02:13.652097 30585 tablet_bootstrap.cc:492] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520: Bootstrap starting.
I20240501 20:02:13.654322 30589 raft_consensus.cc:348] T dfd8f9eb29d745589b4aa63188a0c2aa P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.655036 30589 raft_consensus.cc:374] T dfd8f9eb29d745589b4aa63188a0c2aa P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:13.655323 30589 raft_consensus.cc:729] T dfd8f9eb29d745589b4aa63188a0c2aa P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b430a4ca81472ba0d5c42a4fa6f604, State: Initialized, Role: FOLLOWER
I20240501 20:02:13.656140 30589 consensus_queue.cc:260] T dfd8f9eb29d745589b4aa63188a0c2aa P e4b430a4ca81472ba0d5c42a4fa6f604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.660815 30585 tablet_bootstrap.cc:654] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:13.661563 30539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8bde677f5e4f87802eda8373720358" candidate_uuid: "81885e2958c34da5ae5fe78b749d6e99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "747b41f97e7246cd8a42097546f44520"
I20240501 20:02:13.662348 30539 raft_consensus.cc:3037] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:13.665562 30589 ts_tablet_manager.cc:1398] T dfd8f9eb29d745589b4aa63188a0c2aa P e4b430a4ca81472ba0d5c42a4fa6f604: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20240501 20:02:13.672003 30585 tablet_bootstrap.cc:492] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520: No bootstrap required, opened a new log
I20240501 20:02:13.672583 30585 ts_tablet_manager.cc:1367] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.000s
I20240501 20:02:13.672403 30594 leader_election.cc:288] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [CANDIDATE]: Term 1 election: Requested vote from peers e4b430a4ca81472ba0d5c42a4fa6f604 (127.24.69.130:43685), 747b41f97e7246cd8a42097546f44520 (127.24.69.131:37749)
I20240501 20:02:13.673621 30589 tablet_bootstrap.cc:492] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604: Bootstrap starting.
I20240501 20:02:13.672531 30465 raft_consensus.cc:2454] T 6d8bde677f5e4f87802eda8373720358 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81885e2958c34da5ae5fe78b749d6e99 in term 1.
I20240501 20:02:13.675393 30353 leader_election.cc:302] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81885e2958c34da5ae5fe78b749d6e99, e4b430a4ca81472ba0d5c42a4fa6f604; no voters:
I20240501 20:02:13.675385 30585 raft_consensus.cc:348] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } }
I20240501 20:02:13.672206 30539 raft_consensus.cc:2454] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81885e2958c34da5ae5fe78b749d6e99 in term 1.
I20240501 20:02:13.676632 30585 raft_consensus.cc:374] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:13.677834 30585 raft_consensus.cc:729] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 747b41f97e7246cd8a42097546f44520, State: Initialized, Role: FOLLOWER
I20240501 20:02:13.677042 30594 raft_consensus.cc:2781] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:02:13.678757 30594 raft_consensus.cc:686] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 LEADER]: Becoming Leader. State: Replica: 81885e2958c34da5ae5fe78b749d6e99, State: Running, Role: LEADER
I20240501 20:02:13.679654 30584 tablet_bootstrap.cc:492] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99: No bootstrap required, opened a new log
I20240501 20:02:13.680166 30584 ts_tablet_manager.cc:1367] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99: Time spent bootstrapping tablet: real 0.040s user 0.015s sys 0.021s
I20240501 20:02:13.679590 30594 consensus_queue.cc:237] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.680433 30589 tablet_bootstrap.cc:654] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:13.680544 30585 consensus_queue.cc:260] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } }
I20240501 20:02:13.682933 30585 ts_tablet_manager.cc:1398] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240501 20:02:13.684844 30584 raft_consensus.cc:348] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.685637 30584 raft_consensus.cc:374] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:13.685973 30584 raft_consensus.cc:729] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81885e2958c34da5ae5fe78b749d6e99, State: Initialized, Role: FOLLOWER
I20240501 20:02:13.686710 30584 consensus_queue.cc:260] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.689397 30584 ts_tablet_manager.cc:1398] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240501 20:02:13.692472 30589 tablet_bootstrap.cc:492] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604: No bootstrap required, opened a new log
I20240501 20:02:13.693048 30589 ts_tablet_manager.cc:1367] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.007s
I20240501 20:02:13.694487 30286 catalog_manager.cc:5484] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129). New cstate: current_term: 1 leader_uuid: "81885e2958c34da5ae5fe78b749d6e99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } health_report { overall_health: UNKNOWN } } }
I20240501 20:02:13.695791 30589 raft_consensus.cc:348] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.696583 30589 raft_consensus.cc:374] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:13.696882 30589 raft_consensus.cc:729] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b430a4ca81472ba0d5c42a4fa6f604, State: Initialized, Role: FOLLOWER
I20240501 20:02:13.697557 30589 consensus_queue.cc:260] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.699460 30589 ts_tablet_manager.cc:1398] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240501 20:02:13.705982 30590 raft_consensus.cc:482] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:13.706430 30590 raft_consensus.cc:504] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.708850 30590 leader_election.cc:288] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925), e4b430a4ca81472ba0d5c42a4fa6f604 (127.24.69.130:43685)
I20240501 20:02:13.717880 30390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfd8f9eb29d745589b4aa63188a0c2aa" candidate_uuid: "747b41f97e7246cd8a42097546f44520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81885e2958c34da5ae5fe78b749d6e99" is_pre_election: true
I20240501 20:02:13.718483 30390 raft_consensus.cc:2454] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 747b41f97e7246cd8a42097546f44520 in term 0.
I20240501 20:02:13.719517 30465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfd8f9eb29d745589b4aa63188a0c2aa" candidate_uuid: "747b41f97e7246cd8a42097546f44520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" is_pre_election: true
I20240501 20:02:13.719632 30502 leader_election.cc:302] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 747b41f97e7246cd8a42097546f44520, 81885e2958c34da5ae5fe78b749d6e99; no voters:
I20240501 20:02:13.720038 30465 raft_consensus.cc:2454] T dfd8f9eb29d745589b4aa63188a0c2aa P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 747b41f97e7246cd8a42097546f44520 in term 0.
I20240501 20:02:13.720508 30590 raft_consensus.cc:2781] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:02:13.720868 30590 raft_consensus.cc:482] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:02:13.721125 30590 raft_consensus.cc:3037] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:13.725780 30590 raft_consensus.cc:504] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.727144 30590 leader_election.cc:288] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [CANDIDATE]: Term 1 election: Requested vote from peers 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925), e4b430a4ca81472ba0d5c42a4fa6f604 (127.24.69.130:43685)
I20240501 20:02:13.728013 30390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfd8f9eb29d745589b4aa63188a0c2aa" candidate_uuid: "747b41f97e7246cd8a42097546f44520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81885e2958c34da5ae5fe78b749d6e99"
I20240501 20:02:13.728091 30465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfd8f9eb29d745589b4aa63188a0c2aa" candidate_uuid: "747b41f97e7246cd8a42097546f44520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604"
I20240501 20:02:13.728595 30390 raft_consensus.cc:3037] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:13.728682 30465 raft_consensus.cc:3037] T dfd8f9eb29d745589b4aa63188a0c2aa P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:13.733023 30465 raft_consensus.cc:2454] T dfd8f9eb29d745589b4aa63188a0c2aa P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 747b41f97e7246cd8a42097546f44520 in term 1.
I20240501 20:02:13.733124 30390 raft_consensus.cc:2454] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 747b41f97e7246cd8a42097546f44520 in term 1.
I20240501 20:02:13.733898 30502 leader_election.cc:302] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 747b41f97e7246cd8a42097546f44520, e4b430a4ca81472ba0d5c42a4fa6f604; no voters:
I20240501 20:02:13.734465 30590 raft_consensus.cc:2781] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:02:13.735450 30590 raft_consensus.cc:686] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [term 1 LEADER]: Becoming Leader. State: Replica: 747b41f97e7246cd8a42097546f44520, State: Running, Role: LEADER
I20240501 20:02:13.736047 30590 consensus_queue.cc:237] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.738945 30597 raft_consensus.cc:482] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:13.739441 30597 raft_consensus.cc:504] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.741609 30597 leader_election.cc:288] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925), 747b41f97e7246cd8a42097546f44520 (127.24.69.131:37749)
I20240501 20:02:13.743664 30286 catalog_manager.cc:5484] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 reported cstate change: term changed from 0 to 1, leader changed from <none> to 747b41f97e7246cd8a42097546f44520 (127.24.69.131). New cstate: current_term: 1 leader_uuid: "747b41f97e7246cd8a42097546f44520" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } health_report { overall_health: HEALTHY } } }
I20240501 20:02:13.753150 30390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3fb659b47994126bf3c60689f0b13b5" candidate_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81885e2958c34da5ae5fe78b749d6e99" is_pre_election: true
I20240501 20:02:13.753684 30390 raft_consensus.cc:2454] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4b430a4ca81472ba0d5c42a4fa6f604 in term 0.
I20240501 20:02:13.754446 30539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3fb659b47994126bf3c60689f0b13b5" candidate_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "747b41f97e7246cd8a42097546f44520" is_pre_election: true
I20240501 20:02:13.754597 30428 leader_election.cc:302] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81885e2958c34da5ae5fe78b749d6e99, e4b430a4ca81472ba0d5c42a4fa6f604; no voters:
I20240501 20:02:13.755120 30539 raft_consensus.cc:2454] T d3fb659b47994126bf3c60689f0b13b5 P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4b430a4ca81472ba0d5c42a4fa6f604 in term 0.
I20240501 20:02:13.755357 30597 raft_consensus.cc:2781] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:02:13.755740 30597 raft_consensus.cc:482] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:02:13.756078 30597 raft_consensus.cc:3037] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:13.761402 30597 raft_consensus.cc:504] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.762950 30597 leader_election.cc:288] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [CANDIDATE]: Term 1 election: Requested vote from peers 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925), 747b41f97e7246cd8a42097546f44520 (127.24.69.131:37749)
I20240501 20:02:13.763556 30390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3fb659b47994126bf3c60689f0b13b5" candidate_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81885e2958c34da5ae5fe78b749d6e99"
I20240501 20:02:13.763809 30539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3fb659b47994126bf3c60689f0b13b5" candidate_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "747b41f97e7246cd8a42097546f44520"
I20240501 20:02:13.764042 30390 raft_consensus.cc:3037] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:13.764376 30539 raft_consensus.cc:3037] T d3fb659b47994126bf3c60689f0b13b5 P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:13.768337 30390 raft_consensus.cc:2454] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4b430a4ca81472ba0d5c42a4fa6f604 in term 1.
I20240501 20:02:13.768553 30539 raft_consensus.cc:2454] T d3fb659b47994126bf3c60689f0b13b5 P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4b430a4ca81472ba0d5c42a4fa6f604 in term 1.
I20240501 20:02:13.769253 30428 leader_election.cc:302] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81885e2958c34da5ae5fe78b749d6e99, e4b430a4ca81472ba0d5c42a4fa6f604; no voters:
I20240501 20:02:13.769871 30597 raft_consensus.cc:2781] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:02:13.770895 30597 raft_consensus.cc:686] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 LEADER]: Becoming Leader. State: Replica: e4b430a4ca81472ba0d5c42a4fa6f604, State: Running, Role: LEADER
I20240501 20:02:13.771487 30597 consensus_queue.cc:237] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:13.778103 30286 catalog_manager.cc:5484] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4b430a4ca81472ba0d5c42a4fa6f604 (127.24.69.130). New cstate: current_term: 1 leader_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } health_report { overall_health: UNKNOWN } } }
I20240501 20:02:13.841637 30590 raft_consensus.cc:482] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:13.842108 30590 raft_consensus.cc:504] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } }
I20240501 20:02:13.843644 30590 leader_election.cc:288] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925), e4b430a4ca81472ba0d5c42a4fa6f604 (127.24.69.130:43685)
I20240501 20:02:13.844432 30390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3378dd5a0ac64b3e852cb61ee61cec8b" candidate_uuid: "747b41f97e7246cd8a42097546f44520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81885e2958c34da5ae5fe78b749d6e99" is_pre_election: true
I20240501 20:02:13.844669 30465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3378dd5a0ac64b3e852cb61ee61cec8b" candidate_uuid: "747b41f97e7246cd8a42097546f44520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" is_pre_election: true
I20240501 20:02:13.844988 30390 raft_consensus.cc:2454] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 747b41f97e7246cd8a42097546f44520 in term 0.
I20240501 20:02:13.845234 30465 raft_consensus.cc:2454] T 3378dd5a0ac64b3e852cb61ee61cec8b P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 747b41f97e7246cd8a42097546f44520 in term 0.
I20240501 20:02:13.845997 30502 leader_election.cc:302] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 747b41f97e7246cd8a42097546f44520, 81885e2958c34da5ae5fe78b749d6e99; no voters:
I20240501 20:02:13.846614 30590 raft_consensus.cc:2781] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:02:13.846869 30590 raft_consensus.cc:482] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:02:13.847129 30590 raft_consensus.cc:3037] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:13.851652 30590 raft_consensus.cc:504] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } }
I20240501 20:02:13.853040 30590 leader_election.cc:288] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [CANDIDATE]: Term 1 election: Requested vote from peers 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925), e4b430a4ca81472ba0d5c42a4fa6f604 (127.24.69.130:43685)
I20240501 20:02:13.853878 30390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3378dd5a0ac64b3e852cb61ee61cec8b" candidate_uuid: "747b41f97e7246cd8a42097546f44520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81885e2958c34da5ae5fe78b749d6e99"
I20240501 20:02:13.853999 30465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3378dd5a0ac64b3e852cb61ee61cec8b" candidate_uuid: "747b41f97e7246cd8a42097546f44520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604"
I20240501 20:02:13.854331 30390 raft_consensus.cc:3037] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:13.854516 30465 raft_consensus.cc:3037] T 3378dd5a0ac64b3e852cb61ee61cec8b P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:13.858816 30390 raft_consensus.cc:2454] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 747b41f97e7246cd8a42097546f44520 in term 1.
I20240501 20:02:13.859283 30465 raft_consensus.cc:2454] T 3378dd5a0ac64b3e852cb61ee61cec8b P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 747b41f97e7246cd8a42097546f44520 in term 1.
I20240501 20:02:13.859710 30502 leader_election.cc:302] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 747b41f97e7246cd8a42097546f44520, 81885e2958c34da5ae5fe78b749d6e99; no voters:
I20240501 20:02:13.860414 30590 raft_consensus.cc:2781] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:02:13.860785 30590 raft_consensus.cc:686] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [term 1 LEADER]: Becoming Leader. State: Replica: 747b41f97e7246cd8a42097546f44520, State: Running, Role: LEADER
I20240501 20:02:13.861393 30590 consensus_queue.cc:237] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } }
I20240501 20:02:13.868104 30286 catalog_manager.cc:5484] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 reported cstate change: term changed from 0 to 1, leader changed from <none> to 747b41f97e7246cd8a42097546f44520 (127.24.69.131). New cstate: current_term: 1 leader_uuid: "747b41f97e7246cd8a42097546f44520" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } health_report { overall_health: UNKNOWN } } }
I20240501 20:02:13.923110 24854 test_util.cc:268] Using random seed: -160313075
I20240501 20:02:13.944665 30286 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41500:
name: "dugtrio"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 20:02:13.946849 30286 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table dugtrio in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:02:13.975716 30379 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1f589bca614c0c95b95b4304107784 (DEFAULT_TABLE table=dugtrio [id=c0373a073c56464ca05f378f58035f52]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240501 20:02:13.976981 30379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1f589bca614c0c95b95b4304107784. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:13.977995 30380 tablet_service.cc:1467] Processing CreateTablet for tablet cf264333c5044dd9a9cd2036b1d8d54d (DEFAULT_TABLE table=dugtrio [id=c0373a073c56464ca05f378f58035f52]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240501 20:02:13.978700 30453 tablet_service.cc:1467] Processing CreateTablet for tablet cf264333c5044dd9a9cd2036b1d8d54d (DEFAULT_TABLE table=dugtrio [id=c0373a073c56464ca05f378f58035f52]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240501 20:02:13.979243 30380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf264333c5044dd9a9cd2036b1d8d54d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:13.979907 30453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf264333c5044dd9a9cd2036b1d8d54d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:13.977027 30529 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1f589bca614c0c95b95b4304107784 (DEFAULT_TABLE table=dugtrio [id=c0373a073c56464ca05f378f58035f52]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240501 20:02:13.983299 30529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1f589bca614c0c95b95b4304107784. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:13.984594 30526 tablet_service.cc:1467] Processing CreateTablet for tablet cf264333c5044dd9a9cd2036b1d8d54d (DEFAULT_TABLE table=dugtrio [id=c0373a073c56464ca05f378f58035f52]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240501 20:02:13.985827 30526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf264333c5044dd9a9cd2036b1d8d54d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:13.987023 30452 tablet_service.cc:1467] Processing CreateTablet for tablet e264a1b081524b36822b6b7d95969dba (DEFAULT_TABLE table=dugtrio [id=c0373a073c56464ca05f378f58035f52]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240501 20:02:13.976184 30454 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1f589bca614c0c95b95b4304107784 (DEFAULT_TABLE table=dugtrio [id=c0373a073c56464ca05f378f58035f52]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240501 20:02:13.989091 30455 tablet_service.cc:1467] Processing CreateTablet for tablet 661120bc29cf434e95c3f781d94d9106 (DEFAULT_TABLE table=dugtrio [id=c0373a073c56464ca05f378f58035f52]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240501 20:02:13.989207 30527 tablet_service.cc:1467] Processing CreateTablet for tablet 661120bc29cf434e95c3f781d94d9106 (DEFAULT_TABLE table=dugtrio [id=c0373a073c56464ca05f378f58035f52]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240501 20:02:13.990402 30527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 661120bc29cf434e95c3f781d94d9106. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:13.989105 30528 tablet_service.cc:1467] Processing CreateTablet for tablet e264a1b081524b36822b6b7d95969dba (DEFAULT_TABLE table=dugtrio [id=c0373a073c56464ca05f378f58035f52]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240501 20:02:13.991624 30528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e264a1b081524b36822b6b7d95969dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:13.994450 30377 tablet_service.cc:1467] Processing CreateTablet for tablet e264a1b081524b36822b6b7d95969dba (DEFAULT_TABLE table=dugtrio [id=c0373a073c56464ca05f378f58035f52]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240501 20:02:13.994704 30378 tablet_service.cc:1467] Processing CreateTablet for tablet 661120bc29cf434e95c3f781d94d9106 (DEFAULT_TABLE table=dugtrio [id=c0373a073c56464ca05f378f58035f52]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240501 20:02:13.995659 30377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e264a1b081524b36822b6b7d95969dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:13.996243 30378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 661120bc29cf434e95c3f781d94d9106. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:13.996706 30452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e264a1b081524b36822b6b7d95969dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:14.000569 30454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1f589bca614c0c95b95b4304107784. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:14.001175 30455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 661120bc29cf434e95c3f781d94d9106. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:14.007295 30584 tablet_bootstrap.cc:492] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap starting.
I20240501 20:02:14.013028 30584 tablet_bootstrap.cc:654] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:14.021371 30585 tablet_bootstrap.cc:492] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520: Bootstrap starting.
I20240501 20:02:14.026944 30585 tablet_bootstrap.cc:654] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:14.031518 30589 tablet_bootstrap.cc:492] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604: Bootstrap starting.
I20240501 20:02:14.038053 30589 tablet_bootstrap.cc:654] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:14.044855 30584 tablet_bootstrap.cc:492] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99: No bootstrap required, opened a new log
I20240501 20:02:14.046164 30584 ts_tablet_manager.cc:1367] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.007s
I20240501 20:02:14.049826 30589 tablet_bootstrap.cc:492] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604: No bootstrap required, opened a new log
I20240501 20:02:14.050253 30589 ts_tablet_manager.cc:1367] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.011s
I20240501 20:02:14.050172 30584 raft_consensus.cc:348] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.050798 30584 raft_consensus.cc:374] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:14.051028 30584 raft_consensus.cc:729] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81885e2958c34da5ae5fe78b749d6e99, State: Initialized, Role: FOLLOWER
I20240501 20:02:14.051568 30584 consensus_queue.cc:260] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.052166 30589 raft_consensus.cc:348] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.052892 30589 raft_consensus.cc:374] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:14.053280 30589 raft_consensus.cc:729] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b430a4ca81472ba0d5c42a4fa6f604, State: Initialized, Role: FOLLOWER
I20240501 20:02:14.054123 30589 consensus_queue.cc:260] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.057220 30589 ts_tablet_manager.cc:1398] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240501 20:02:14.058354 30589 tablet_bootstrap.cc:492] T e264a1b081524b36822b6b7d95969dba P e4b430a4ca81472ba0d5c42a4fa6f604: Bootstrap starting.
I20240501 20:02:14.060482 30585 tablet_bootstrap.cc:492] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520: No bootstrap required, opened a new log
I20240501 20:02:14.061031 30585 ts_tablet_manager.cc:1367] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.007s
I20240501 20:02:14.063495 30589 tablet_bootstrap.cc:654] T e264a1b081524b36822b6b7d95969dba P e4b430a4ca81472ba0d5c42a4fa6f604: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:14.067006 30585 raft_consensus.cc:348] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.068161 30585 raft_consensus.cc:374] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:14.068603 30585 raft_consensus.cc:729] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 747b41f97e7246cd8a42097546f44520, State: Initialized, Role: FOLLOWER
I20240501 20:02:14.073002 30585 consensus_queue.cc:260] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.073551 30584 ts_tablet_manager.cc:1398] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99: Time spent starting tablet: real 0.025s user 0.008s sys 0.008s
I20240501 20:02:14.074992 30584 tablet_bootstrap.cc:492] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap starting.
I20240501 20:02:14.076650 30585 ts_tablet_manager.cc:1398] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20240501 20:02:14.077672 30585 tablet_bootstrap.cc:492] T 661120bc29cf434e95c3f781d94d9106 P 747b41f97e7246cd8a42097546f44520: Bootstrap starting.
I20240501 20:02:14.082492 30584 tablet_bootstrap.cc:654] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:14.083534 30585 tablet_bootstrap.cc:654] T 661120bc29cf434e95c3f781d94d9106 P 747b41f97e7246cd8a42097546f44520: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:14.076364 30589 tablet_bootstrap.cc:492] T e264a1b081524b36822b6b7d95969dba P e4b430a4ca81472ba0d5c42a4fa6f604: No bootstrap required, opened a new log
I20240501 20:02:14.085654 30589 ts_tablet_manager.cc:1367] T e264a1b081524b36822b6b7d95969dba P e4b430a4ca81472ba0d5c42a4fa6f604: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.004s
I20240501 20:02:14.088290 30589 raft_consensus.cc:348] T e264a1b081524b36822b6b7d95969dba P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.089031 30589 raft_consensus.cc:374] T e264a1b081524b36822b6b7d95969dba P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:14.089318 30589 raft_consensus.cc:729] T e264a1b081524b36822b6b7d95969dba P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b430a4ca81472ba0d5c42a4fa6f604, State: Initialized, Role: FOLLOWER
I20240501 20:02:14.089967 30590 raft_consensus.cc:482] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:14.090230 30584 tablet_bootstrap.cc:492] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99: No bootstrap required, opened a new log
I20240501 20:02:14.090054 30589 consensus_queue.cc:260] T e264a1b081524b36822b6b7d95969dba P e4b430a4ca81472ba0d5c42a4fa6f604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.090602 30585 tablet_bootstrap.cc:492] T 661120bc29cf434e95c3f781d94d9106 P 747b41f97e7246cd8a42097546f44520: No bootstrap required, opened a new log
I20240501 20:02:14.090742 30584 ts_tablet_manager.cc:1367] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.004s
I20240501 20:02:14.090550 30590 raft_consensus.cc:504] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.091277 30585 ts_tablet_manager.cc:1367] T 661120bc29cf434e95c3f781d94d9106 P 747b41f97e7246cd8a42097546f44520: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20240501 20:02:14.093379 30390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf264333c5044dd9a9cd2036b1d8d54d" candidate_uuid: "747b41f97e7246cd8a42097546f44520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81885e2958c34da5ae5fe78b749d6e99" is_pre_election: true
I20240501 20:02:14.093578 30584 raft_consensus.cc:348] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.094280 30584 raft_consensus.cc:374] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:14.093991 30585 raft_consensus.cc:348] T 661120bc29cf434e95c3f781d94d9106 P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.094606 30584 raft_consensus.cc:729] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81885e2958c34da5ae5fe78b749d6e99, State: Initialized, Role: FOLLOWER
I20240501 20:02:14.094763 30585 raft_consensus.cc:374] T 661120bc29cf434e95c3f781d94d9106 P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:14.095113 30585 raft_consensus.cc:729] T 661120bc29cf434e95c3f781d94d9106 P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 747b41f97e7246cd8a42097546f44520, State: Initialized, Role: FOLLOWER
I20240501 20:02:14.095223 30589 ts_tablet_manager.cc:1398] T e264a1b081524b36822b6b7d95969dba P e4b430a4ca81472ba0d5c42a4fa6f604: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240501 20:02:14.095395 30584 consensus_queue.cc:260] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.096299 30589 tablet_bootstrap.cc:492] T 2b1f589bca614c0c95b95b4304107784 P e4b430a4ca81472ba0d5c42a4fa6f604: Bootstrap starting.
I20240501 20:02:14.097790 30584 ts_tablet_manager.cc:1398] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20240501 20:02:14.098740 30584 tablet_bootstrap.cc:492] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap starting.
I20240501 20:02:14.102646 30589 tablet_bootstrap.cc:654] T 2b1f589bca614c0c95b95b4304107784 P e4b430a4ca81472ba0d5c42a4fa6f604: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:14.104277 30584 tablet_bootstrap.cc:654] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:14.105340 30390 raft_consensus.cc:2454] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 747b41f97e7246cd8a42097546f44520 in term 0.
I20240501 20:02:14.107379 30502 leader_election.cc:302] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 747b41f97e7246cd8a42097546f44520, 81885e2958c34da5ae5fe78b749d6e99; no voters:
I20240501 20:02:14.108513 30608 raft_consensus.cc:2781] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:02:14.108976 30608 raft_consensus.cc:482] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:02:14.109367 30608 raft_consensus.cc:3037] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:14.095886 30585 consensus_queue.cc:260] T 661120bc29cf434e95c3f781d94d9106 P 747b41f97e7246cd8a42097546f44520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.116744 30597 raft_consensus.cc:482] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:14.117481 30590 leader_election.cc:288] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925), e4b430a4ca81472ba0d5c42a4fa6f604 (127.24.69.130:43685)
I20240501 20:02:14.117386 30597 raft_consensus.cc:504] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.119675 30465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf264333c5044dd9a9cd2036b1d8d54d" candidate_uuid: "747b41f97e7246cd8a42097546f44520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" is_pre_election: true
I20240501 20:02:14.120544 30465 raft_consensus.cc:2454] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 747b41f97e7246cd8a42097546f44520 in term 0.
I20240501 20:02:14.121328 30597 leader_election.cc:288] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925), 747b41f97e7246cd8a42097546f44520 (127.24.69.131:37749)
I20240501 20:02:14.128340 30608 raft_consensus.cc:504] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.130964 30608 leader_election.cc:288] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520 [CANDIDATE]: Term 1 election: Requested vote from peers 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925), e4b430a4ca81472ba0d5c42a4fa6f604 (127.24.69.130:43685)
I20240501 20:02:14.131237 30585 ts_tablet_manager.cc:1398] T 661120bc29cf434e95c3f781d94d9106 P 747b41f97e7246cd8a42097546f44520: Time spent starting tablet: real 0.040s user 0.010s sys 0.000s
I20240501 20:02:14.131446 30390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf264333c5044dd9a9cd2036b1d8d54d" candidate_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81885e2958c34da5ae5fe78b749d6e99" is_pre_election: true
I20240501 20:02:14.132143 30390 raft_consensus.cc:2454] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4b430a4ca81472ba0d5c42a4fa6f604 in term 0.
I20240501 20:02:14.132676 30585 tablet_bootstrap.cc:492] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520: Bootstrap starting.
I20240501 20:02:14.133251 30428 leader_election.cc:302] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81885e2958c34da5ae5fe78b749d6e99, e4b430a4ca81472ba0d5c42a4fa6f604; no voters:
I20240501 20:02:14.134074 30597 raft_consensus.cc:2781] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:02:14.134356 30597 raft_consensus.cc:482] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:02:14.134616 30597 raft_consensus.cc:3037] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:14.140537 30585 tablet_bootstrap.cc:654] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:14.140741 30539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf264333c5044dd9a9cd2036b1d8d54d" candidate_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "747b41f97e7246cd8a42097546f44520" is_pre_election: true
I20240501 20:02:14.141597 30539 raft_consensus.cc:2379] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4b430a4ca81472ba0d5c42a4fa6f604 in current term 1: Already voted for candidate 747b41f97e7246cd8a42097546f44520 in this term.
I20240501 20:02:14.141790 30389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf264333c5044dd9a9cd2036b1d8d54d" candidate_uuid: "747b41f97e7246cd8a42097546f44520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81885e2958c34da5ae5fe78b749d6e99"
I20240501 20:02:14.142438 30389 raft_consensus.cc:3037] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:14.142864 30465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf264333c5044dd9a9cd2036b1d8d54d" candidate_uuid: "747b41f97e7246cd8a42097546f44520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604"
I20240501 20:02:14.144665 30597 raft_consensus.cc:504] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.152046 30465 raft_consensus.cc:2379] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 747b41f97e7246cd8a42097546f44520 in current term 1: Already voted for candidate e4b430a4ca81472ba0d5c42a4fa6f604 in this term.
I20240501 20:02:14.153483 30390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf264333c5044dd9a9cd2036b1d8d54d" candidate_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81885e2958c34da5ae5fe78b749d6e99"
I20240501 20:02:14.155651 30589 tablet_bootstrap.cc:492] T 2b1f589bca614c0c95b95b4304107784 P e4b430a4ca81472ba0d5c42a4fa6f604: No bootstrap required, opened a new log
I20240501 20:02:14.155951 30597 leader_election.cc:288] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604 [CANDIDATE]: Term 1 election: Requested vote from peers 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925), 747b41f97e7246cd8a42097546f44520 (127.24.69.131:37749)
I20240501 20:02:14.156181 30589 ts_tablet_manager.cc:1367] T 2b1f589bca614c0c95b95b4304107784 P e4b430a4ca81472ba0d5c42a4fa6f604: Time spent bootstrapping tablet: real 0.060s user 0.019s sys 0.012s
I20240501 20:02:14.157311 30539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf264333c5044dd9a9cd2036b1d8d54d" candidate_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "747b41f97e7246cd8a42097546f44520"
I20240501 20:02:14.157804 30585 tablet_bootstrap.cc:492] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520: No bootstrap required, opened a new log
I20240501 20:02:14.157691 30584 tablet_bootstrap.cc:492] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99: No bootstrap required, opened a new log
I20240501 20:02:14.158134 30539 raft_consensus.cc:2379] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e4b430a4ca81472ba0d5c42a4fa6f604 in current term 1: Already voted for candidate 747b41f97e7246cd8a42097546f44520 in this term.
I20240501 20:02:14.158484 30585 ts_tablet_manager.cc:1367] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.003s
I20240501 20:02:14.158587 30584 ts_tablet_manager.cc:1367] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99: Time spent bootstrapping tablet: real 0.060s user 0.016s sys 0.032s
I20240501 20:02:14.159082 30589 raft_consensus.cc:348] T 2b1f589bca614c0c95b95b4304107784 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.159758 30429 leader_election.cc:302] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4b430a4ca81472ba0d5c42a4fa6f604; no voters: 747b41f97e7246cd8a42097546f44520, 81885e2958c34da5ae5fe78b749d6e99
I20240501 20:02:14.159946 30589 raft_consensus.cc:374] T 2b1f589bca614c0c95b95b4304107784 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:14.160434 30589 raft_consensus.cc:729] T 2b1f589bca614c0c95b95b4304107784 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b430a4ca81472ba0d5c42a4fa6f604, State: Initialized, Role: FOLLOWER
I20240501 20:02:14.161152 30597 raft_consensus.cc:2726] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 20:02:14.154976 30389 raft_consensus.cc:2454] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 747b41f97e7246cd8a42097546f44520 in term 1.
I20240501 20:02:14.161314 30589 consensus_queue.cc:260] T 2b1f589bca614c0c95b95b4304107784 P e4b430a4ca81472ba0d5c42a4fa6f604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.161254 30585 raft_consensus.cc:348] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.162375 30585 raft_consensus.cc:374] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:14.162412 30502 leader_election.cc:302] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 747b41f97e7246cd8a42097546f44520, 81885e2958c34da5ae5fe78b749d6e99; no voters: e4b430a4ca81472ba0d5c42a4fa6f604
I20240501 20:02:14.162789 30585 raft_consensus.cc:729] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 747b41f97e7246cd8a42097546f44520, State: Initialized, Role: FOLLOWER
I20240501 20:02:14.163322 30608 raft_consensus.cc:2781] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:02:14.163769 30589 ts_tablet_manager.cc:1398] T 2b1f589bca614c0c95b95b4304107784 P e4b430a4ca81472ba0d5c42a4fa6f604: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20240501 20:02:14.163836 30608 raft_consensus.cc:686] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520 [term 1 LEADER]: Becoming Leader. State: Replica: 747b41f97e7246cd8a42097546f44520, State: Running, Role: LEADER
I20240501 20:02:14.164848 30589 tablet_bootstrap.cc:492] T 661120bc29cf434e95c3f781d94d9106 P e4b430a4ca81472ba0d5c42a4fa6f604: Bootstrap starting.
I20240501 20:02:14.163614 30585 consensus_queue.cc:260] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.164741 30608 consensus_queue.cc:237] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.161892 30584 raft_consensus.cc:348] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.165936 30584 raft_consensus.cc:374] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:14.166293 30584 raft_consensus.cc:729] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81885e2958c34da5ae5fe78b749d6e99, State: Initialized, Role: FOLLOWER
I20240501 20:02:14.167007 30585 ts_tablet_manager.cc:1398] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520: Time spent starting tablet: real 0.008s user 0.002s sys 0.005s
I20240501 20:02:14.167050 30584 consensus_queue.cc:260] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.168135 30585 tablet_bootstrap.cc:492] T 2b1f589bca614c0c95b95b4304107784 P 747b41f97e7246cd8a42097546f44520: Bootstrap starting.
I20240501 20:02:14.169931 30584 ts_tablet_manager.cc:1398] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99: Time spent starting tablet: real 0.011s user 0.003s sys 0.002s
I20240501 20:02:14.174474 30584 tablet_bootstrap.cc:492] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap starting.
I20240501 20:02:14.176518 30589 tablet_bootstrap.cc:654] T 661120bc29cf434e95c3f781d94d9106 P e4b430a4ca81472ba0d5c42a4fa6f604: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:14.183007 30584 tablet_bootstrap.cc:654] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:14.183912 30287 catalog_manager.cc:5484] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520 reported cstate change: term changed from 0 to 1, leader changed from <none> to 747b41f97e7246cd8a42097546f44520 (127.24.69.131). New cstate: current_term: 1 leader_uuid: "747b41f97e7246cd8a42097546f44520" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } health_report { overall_health: HEALTHY } } }
I20240501 20:02:14.190997 30585 tablet_bootstrap.cc:654] T 2b1f589bca614c0c95b95b4304107784 P 747b41f97e7246cd8a42097546f44520: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:14.199839 30585 tablet_bootstrap.cc:492] T 2b1f589bca614c0c95b95b4304107784 P 747b41f97e7246cd8a42097546f44520: No bootstrap required, opened a new log
I20240501 20:02:14.199860 30584 tablet_bootstrap.cc:492] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99: No bootstrap required, opened a new log
I20240501 20:02:14.200527 30584 ts_tablet_manager.cc:1367] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.013s
I20240501 20:02:14.200496 30585 ts_tablet_manager.cc:1367] T 2b1f589bca614c0c95b95b4304107784 P 747b41f97e7246cd8a42097546f44520: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.011s
I20240501 20:02:14.202746 30589 tablet_bootstrap.cc:492] T 661120bc29cf434e95c3f781d94d9106 P e4b430a4ca81472ba0d5c42a4fa6f604: No bootstrap required, opened a new log
I20240501 20:02:14.203290 30589 ts_tablet_manager.cc:1367] T 661120bc29cf434e95c3f781d94d9106 P e4b430a4ca81472ba0d5c42a4fa6f604: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.020s
I20240501 20:02:14.203198 30584 raft_consensus.cc:348] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.203344 30585 raft_consensus.cc:348] T 2b1f589bca614c0c95b95b4304107784 P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.204001 30584 raft_consensus.cc:374] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:14.204098 30585 raft_consensus.cc:374] T 2b1f589bca614c0c95b95b4304107784 P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:14.204444 30584 raft_consensus.cc:729] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81885e2958c34da5ae5fe78b749d6e99, State: Initialized, Role: FOLLOWER
I20240501 20:02:14.204530 30585 raft_consensus.cc:729] T 2b1f589bca614c0c95b95b4304107784 P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 747b41f97e7246cd8a42097546f44520, State: Initialized, Role: FOLLOWER
I20240501 20:02:14.205217 30584 consensus_queue.cc:260] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.205389 30585 consensus_queue.cc:260] T 2b1f589bca614c0c95b95b4304107784 P 747b41f97e7246cd8a42097546f44520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.206238 30589 raft_consensus.cc:348] T 661120bc29cf434e95c3f781d94d9106 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.206885 30589 raft_consensus.cc:374] T 661120bc29cf434e95c3f781d94d9106 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:14.207134 30589 raft_consensus.cc:729] T 661120bc29cf434e95c3f781d94d9106 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b430a4ca81472ba0d5c42a4fa6f604, State: Initialized, Role: FOLLOWER
I20240501 20:02:14.207406 30585 ts_tablet_manager.cc:1398] T 2b1f589bca614c0c95b95b4304107784 P 747b41f97e7246cd8a42097546f44520: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 20:02:14.207808 30589 consensus_queue.cc:260] T 661120bc29cf434e95c3f781d94d9106 P e4b430a4ca81472ba0d5c42a4fa6f604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.209290 30590 consensus_queue.cc:1035] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:02:14.209746 30589 ts_tablet_manager.cc:1398] T 661120bc29cf434e95c3f781d94d9106 P e4b430a4ca81472ba0d5c42a4fa6f604: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:02:14.210069 30584 ts_tablet_manager.cc:1398] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240501 20:02:14.237363 30590 consensus_queue.cc:1035] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:02:14.239511 30594 consensus_queue.cc:1035] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:02:14.256089 30642 consensus_queue.cc:1035] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:02:14.256537 30594 raft_consensus.cc:482] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:14.257041 30594 raft_consensus.cc:504] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.258952 30594 leader_election.cc:288] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4b430a4ca81472ba0d5c42a4fa6f604 (127.24.69.130:43685), 747b41f97e7246cd8a42097546f44520 (127.24.69.131:37749)
I20240501 20:02:14.259299 30590 consensus_queue.cc:1035] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:02:14.260113 30538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1f589bca614c0c95b95b4304107784" candidate_uuid: "81885e2958c34da5ae5fe78b749d6e99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "747b41f97e7246cd8a42097546f44520" is_pre_election: true
I20240501 20:02:14.260802 30538 raft_consensus.cc:2454] T 2b1f589bca614c0c95b95b4304107784 P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81885e2958c34da5ae5fe78b749d6e99 in term 0.
I20240501 20:02:14.262063 30354 leader_election.cc:302] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 747b41f97e7246cd8a42097546f44520, 81885e2958c34da5ae5fe78b749d6e99; no voters:
I20240501 20:02:14.262909 30594 raft_consensus.cc:2781] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:02:14.263260 30594 raft_consensus.cc:482] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:02:14.263586 30594 raft_consensus.cc:3037] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:14.268661 30463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1f589bca614c0c95b95b4304107784" candidate_uuid: "81885e2958c34da5ae5fe78b749d6e99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" is_pre_election: true
I20240501 20:02:14.269373 30463 raft_consensus.cc:2454] T 2b1f589bca614c0c95b95b4304107784 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81885e2958c34da5ae5fe78b749d6e99 in term 0.
I20240501 20:02:14.272912 30594 raft_consensus.cc:504] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.276721 30463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1f589bca614c0c95b95b4304107784" candidate_uuid: "81885e2958c34da5ae5fe78b749d6e99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604"
I20240501 20:02:14.277441 30463 raft_consensus.cc:3037] T 2b1f589bca614c0c95b95b4304107784 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:14.278827 30594 leader_election.cc:288] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 [CANDIDATE]: Term 1 election: Requested vote from peers e4b430a4ca81472ba0d5c42a4fa6f604 (127.24.69.130:43685), 747b41f97e7246cd8a42097546f44520 (127.24.69.131:37749)
I20240501 20:02:14.280668 30539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1f589bca614c0c95b95b4304107784" candidate_uuid: "81885e2958c34da5ae5fe78b749d6e99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "747b41f97e7246cd8a42097546f44520"
I20240501 20:02:14.281409 30539 raft_consensus.cc:3037] T 2b1f589bca614c0c95b95b4304107784 P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:14.286247 30463 raft_consensus.cc:2454] T 2b1f589bca614c0c95b95b4304107784 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81885e2958c34da5ae5fe78b749d6e99 in term 1.
I20240501 20:02:14.287433 30353 leader_election.cc:302] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81885e2958c34da5ae5fe78b749d6e99, e4b430a4ca81472ba0d5c42a4fa6f604; no voters:
I20240501 20:02:14.287853 30539 raft_consensus.cc:2454] T 2b1f589bca614c0c95b95b4304107784 P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81885e2958c34da5ae5fe78b749d6e99 in term 1.
I20240501 20:02:14.288404 30594 raft_consensus.cc:2781] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:02:14.290189 30594 raft_consensus.cc:686] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 LEADER]: Becoming Leader. State: Replica: 81885e2958c34da5ae5fe78b749d6e99, State: Running, Role: LEADER
I20240501 20:02:14.290961 30594 consensus_queue.cc:237] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.293512 30612 consensus_queue.cc:1035] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:02:14.301944 30612 consensus_queue.cc:1035] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:02:14.306048 30629 consensus_queue.cc:1035] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:02:14.312539 30287 catalog_manager.cc:5484] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129). New cstate: current_term: 1 leader_uuid: "81885e2958c34da5ae5fe78b749d6e99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } health_report { overall_health: UNKNOWN } } }
I20240501 20:02:14.400563 30608 raft_consensus.cc:482] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:14.400974 30608 raft_consensus.cc:504] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.402416 30608 leader_election.cc:288] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925), e4b430a4ca81472ba0d5c42a4fa6f604 (127.24.69.130:43685)
I20240501 20:02:14.403292 30390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e264a1b081524b36822b6b7d95969dba" candidate_uuid: "747b41f97e7246cd8a42097546f44520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81885e2958c34da5ae5fe78b749d6e99" is_pre_election: true
I20240501 20:02:14.403441 30465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e264a1b081524b36822b6b7d95969dba" candidate_uuid: "747b41f97e7246cd8a42097546f44520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" is_pre_election: true
I20240501 20:02:14.403841 30390 raft_consensus.cc:2454] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 747b41f97e7246cd8a42097546f44520 in term 0.
I20240501 20:02:14.404031 30465 raft_consensus.cc:2454] T e264a1b081524b36822b6b7d95969dba P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 747b41f97e7246cd8a42097546f44520 in term 0.
I20240501 20:02:14.404747 30502 leader_election.cc:302] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 747b41f97e7246cd8a42097546f44520, 81885e2958c34da5ae5fe78b749d6e99; no voters:
I20240501 20:02:14.405439 30608 raft_consensus.cc:2781] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:02:14.405779 30608 raft_consensus.cc:482] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:02:14.406064 30608 raft_consensus.cc:3037] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:14.410669 30608 raft_consensus.cc:504] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.412151 30608 leader_election.cc:288] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520 [CANDIDATE]: Term 1 election: Requested vote from peers 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925), e4b430a4ca81472ba0d5c42a4fa6f604 (127.24.69.130:43685)
I20240501 20:02:14.412900 30390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e264a1b081524b36822b6b7d95969dba" candidate_uuid: "747b41f97e7246cd8a42097546f44520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81885e2958c34da5ae5fe78b749d6e99"
I20240501 20:02:14.413115 30465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e264a1b081524b36822b6b7d95969dba" candidate_uuid: "747b41f97e7246cd8a42097546f44520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604"
I20240501 20:02:14.413381 30390 raft_consensus.cc:3037] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:14.413609 30465 raft_consensus.cc:3037] T e264a1b081524b36822b6b7d95969dba P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:14.416153 30594 raft_consensus.cc:482] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:14.416639 30594 raft_consensus.cc:504] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.418380 30390 raft_consensus.cc:2454] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 747b41f97e7246cd8a42097546f44520 in term 1.
I20240501 20:02:14.418612 30594 leader_election.cc:288] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4b430a4ca81472ba0d5c42a4fa6f604 (127.24.69.130:43685), 747b41f97e7246cd8a42097546f44520 (127.24.69.131:37749)
I20240501 20:02:14.419088 30465 raft_consensus.cc:2454] T e264a1b081524b36822b6b7d95969dba P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 747b41f97e7246cd8a42097546f44520 in term 1.
I20240501 20:02:14.419385 30463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "661120bc29cf434e95c3f781d94d9106" candidate_uuid: "81885e2958c34da5ae5fe78b749d6e99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" is_pre_election: true
I20240501 20:02:14.419935 30502 leader_election.cc:302] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 747b41f97e7246cd8a42097546f44520, 81885e2958c34da5ae5fe78b749d6e99; no voters:
I20240501 20:02:14.420034 30463 raft_consensus.cc:2454] T 661120bc29cf434e95c3f781d94d9106 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81885e2958c34da5ae5fe78b749d6e99 in term 0.
I20240501 20:02:14.420212 30539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "661120bc29cf434e95c3f781d94d9106" candidate_uuid: "81885e2958c34da5ae5fe78b749d6e99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "747b41f97e7246cd8a42097546f44520" is_pre_election: true
I20240501 20:02:14.420827 30608 raft_consensus.cc:2781] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:02:14.420852 30539 raft_consensus.cc:2454] T 661120bc29cf434e95c3f781d94d9106 P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81885e2958c34da5ae5fe78b749d6e99 in term 0.
I20240501 20:02:14.421240 30353 leader_election.cc:302] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81885e2958c34da5ae5fe78b749d6e99, e4b430a4ca81472ba0d5c42a4fa6f604; no voters:
I20240501 20:02:14.422087 30594 raft_consensus.cc:2781] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:02:14.422257 30608 raft_consensus.cc:686] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520 [term 1 LEADER]: Becoming Leader. State: Replica: 747b41f97e7246cd8a42097546f44520, State: Running, Role: LEADER
I20240501 20:02:14.422370 30594 raft_consensus.cc:482] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:02:14.422756 30594 raft_consensus.cc:3037] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:14.423053 30608 consensus_queue.cc:237] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.428002 30594 raft_consensus.cc:504] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.429750 30594 leader_election.cc:288] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 [CANDIDATE]: Term 1 election: Requested vote from peers e4b430a4ca81472ba0d5c42a4fa6f604 (127.24.69.130:43685), 747b41f97e7246cd8a42097546f44520 (127.24.69.131:37749)
I20240501 20:02:14.430522 30463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "661120bc29cf434e95c3f781d94d9106" candidate_uuid: "81885e2958c34da5ae5fe78b749d6e99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604"
I20240501 20:02:14.431123 30463 raft_consensus.cc:3037] T 661120bc29cf434e95c3f781d94d9106 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:14.431596 30539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "661120bc29cf434e95c3f781d94d9106" candidate_uuid: "81885e2958c34da5ae5fe78b749d6e99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "747b41f97e7246cd8a42097546f44520"
I20240501 20:02:14.432124 30539 raft_consensus.cc:3037] T 661120bc29cf434e95c3f781d94d9106 P 747b41f97e7246cd8a42097546f44520 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:14.431519 30285 catalog_manager.cc:5484] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520 reported cstate change: term changed from 0 to 1, leader changed from <none> to 747b41f97e7246cd8a42097546f44520 (127.24.69.131). New cstate: current_term: 1 leader_uuid: "747b41f97e7246cd8a42097546f44520" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } health_report { overall_health: HEALTHY } } }
I20240501 20:02:14.436614 30463 raft_consensus.cc:2454] T 661120bc29cf434e95c3f781d94d9106 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81885e2958c34da5ae5fe78b749d6e99 in term 1.
I20240501 20:02:14.437958 30539 raft_consensus.cc:2454] T 661120bc29cf434e95c3f781d94d9106 P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81885e2958c34da5ae5fe78b749d6e99 in term 1.
I20240501 20:02:14.438053 30353 leader_election.cc:302] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81885e2958c34da5ae5fe78b749d6e99, e4b430a4ca81472ba0d5c42a4fa6f604; no voters:
I20240501 20:02:14.438879 30594 raft_consensus.cc:2781] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:02:14.439292 30594 raft_consensus.cc:686] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 LEADER]: Becoming Leader. State: Replica: 81885e2958c34da5ae5fe78b749d6e99, State: Running, Role: LEADER
I20240501 20:02:14.440016 30594 consensus_queue.cc:237] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:14.447080 30285 catalog_manager.cc:5484] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129). New cstate: current_term: 1 leader_uuid: "81885e2958c34da5ae5fe78b749d6e99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } health_report { overall_health: UNKNOWN } } }
I20240501 20:02:14.485808 24854 tablet_server.cc:178] TabletServer@127.24.69.129:0 shutting down...
I20240501 20:02:14.524231 24854 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:02:14.525070 24854 tablet_replica.cc:330] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99: stopping tablet replica
I20240501 20:02:14.525712 24854 raft_consensus.cc:2229] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:02:14.526268 24854 pending_rounds.cc:62] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99: Trying to abort 1 pending ops.
I20240501 20:02:14.526430 24854 pending_rounds.cc:68] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022975936273108992 op_type: NO_OP noop_request { }
I20240501 20:02:14.526669 24854 raft_consensus.cc:2866] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240501 20:02:14.526888 24854 raft_consensus.cc:2258] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:14.528924 24854 tablet_replica.cc:330] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99: stopping tablet replica
I20240501 20:02:14.529403 24854 raft_consensus.cc:2229] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:14.529706 24854 raft_consensus.cc:2258] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:14.531312 24854 tablet_replica.cc:330] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99: stopping tablet replica
I20240501 20:02:14.531764 24854 raft_consensus.cc:2229] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:14.532049 24854 raft_consensus.cc:2258] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:14.533598 24854 tablet_replica.cc:330] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99: stopping tablet replica
I20240501 20:02:14.534036 24854 raft_consensus.cc:2229] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:14.534358 24854 raft_consensus.cc:2258] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:14.536077 24854 tablet_replica.cc:330] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99: stopping tablet replica
I20240501 20:02:14.536559 24854 raft_consensus.cc:2229] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:14.536880 24854 raft_consensus.cc:2258] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:14.538766 24854 tablet_replica.cc:330] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99: stopping tablet replica
I20240501 20:02:14.539242 24854 raft_consensus.cc:2229] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:02:14.539966 24854 raft_consensus.cc:2258] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:14.541798 24854 tablet_replica.cc:330] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99: stopping tablet replica
I20240501 20:02:14.542271 24854 raft_consensus.cc:2229] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:02:14.542732 24854 pending_rounds.cc:62] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99: Trying to abort 1 pending ops.
I20240501 20:02:14.542876 24854 pending_rounds.cc:68] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022975935664107520 op_type: NO_OP noop_request { }
I20240501 20:02:14.543129 24854 raft_consensus.cc:2866] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240501 20:02:14.543407 24854 raft_consensus.cc:2258] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:14.545224 24854 tablet_replica.cc:330] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99: stopping tablet replica
I20240501 20:02:14.545620 24854 raft_consensus.cc:2229] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:14.545990 24854 raft_consensus.cc:2258] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:14.562438 24854 tablet_server.cc:194] TabletServer@127.24.69.129:0 shutdown complete.
I20240501 20:02:14.567680 30608 consensus_queue.cc:1035] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:02:14.584900 30502 consensus_peers.cc:485] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520 -> Peer 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925): Couldn't send request to peer 81885e2958c34da5ae5fe78b749d6e99. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:35925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:02:14.617846 30502 consensus_peers.cc:485] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 -> Peer 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925): Couldn't send request to peer 81885e2958c34da5ae5fe78b749d6e99. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:35925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:02:14.753741 30502 consensus_peers.cc:485] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 -> Peer 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925): Couldn't send request to peer 81885e2958c34da5ae5fe78b749d6e99. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:35925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:02:14.815961 30428 consensus_peers.cc:485] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 -> Peer 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925): Couldn't send request to peer 81885e2958c34da5ae5fe78b749d6e99. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:35925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:02:15.014245 30502 consensus_peers.cc:485] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520 -> Peer 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925): Couldn't send request to peer 81885e2958c34da5ae5fe78b749d6e99. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:35925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:02:15.034729 30629 consensus_queue.cc:1035] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:02:15.413373 30650 consensus_queue.cc:579] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [LEADER]: Leader has been unable to successfully communicate with peer 81885e2958c34da5ae5fe78b749d6e99 for more than 1 seconds (1.092s)
I20240501 20:02:15.505437 30659 consensus_queue.cc:579] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520 [LEADER]: Leader has been unable to successfully communicate with peer 81885e2958c34da5ae5fe78b749d6e99 for more than 1 seconds (1.339s)
I20240501 20:02:15.547776 30659 consensus_queue.cc:579] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520 [LEADER]: Leader has been unable to successfully communicate with peer 81885e2958c34da5ae5fe78b749d6e99 for more than 1 seconds (1.124s)
I20240501 20:02:15.594480 30285 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41498:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20240501 20:02:15.594990 30285 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41498:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20240501 20:02:15.609400 30285 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1191fc0bfcfd4bb9b091152625e0abb4: Sending DeleteTablet for 3 replicas of tablet 2b1f589bca614c0c95b95b4304107784
I20240501 20:02:15.611315 30285 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1191fc0bfcfd4bb9b091152625e0abb4: Sending DeleteTablet for 3 replicas of tablet cf264333c5044dd9a9cd2036b1d8d54d
I20240501 20:02:15.612370 30529 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1f589bca614c0c95b95b4304107784 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:02:15 UTC) from {username='slave'} at 127.0.0.1:37634
I20240501 20:02:15.613211 30455 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1f589bca614c0c95b95b4304107784 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:02:15 UTC) from {username='slave'} at 127.0.0.1:56520
I20240501 20:02:15.614490 30528 tablet_service.cc:1514] Processing DeleteTablet for tablet cf264333c5044dd9a9cd2036b1d8d54d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:02:15 UTC) from {username='slave'} at 127.0.0.1:37634
I20240501 20:02:15.614460 30454 tablet_service.cc:1514] Processing DeleteTablet for tablet cf264333c5044dd9a9cd2036b1d8d54d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:02:15 UTC) from {username='slave'} at 127.0.0.1:56520
I20240501 20:02:15.615190 30659 consensus_queue.cc:579] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [LEADER]: Leader has been unable to successfully communicate with peer 81885e2958c34da5ae5fe78b749d6e99 for more than 1 seconds (1.291s)
I20240501 20:02:15.613607 30285 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1191fc0bfcfd4bb9b091152625e0abb4: Sending DeleteTablet for 3 replicas of tablet e264a1b081524b36822b6b7d95969dba
I20240501 20:02:15.618587 30528 tablet_service.cc:1514] Processing DeleteTablet for tablet e264a1b081524b36822b6b7d95969dba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:02:15 UTC) from {username='slave'} at 127.0.0.1:37634
I20240501 20:02:15.619379 30285 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1191fc0bfcfd4bb9b091152625e0abb4: Sending DeleteTablet for 3 replicas of tablet 661120bc29cf434e95c3f781d94d9106
I20240501 20:02:15.622473 30528 tablet_service.cc:1514] Processing DeleteTablet for tablet 661120bc29cf434e95c3f781d94d9106 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:02:15 UTC) from {username='slave'} at 127.0.0.1:37634
I20240501 20:02:15.624712 30672 tablet_replica.cc:330] T 2b1f589bca614c0c95b95b4304107784 P e4b430a4ca81472ba0d5c42a4fa6f604: stopping tablet replica
I20240501 20:02:15.625119 30671 tablet_replica.cc:330] T 2b1f589bca614c0c95b95b4304107784 P 747b41f97e7246cd8a42097546f44520: stopping tablet replica
I20240501 20:02:15.625722 30672 raft_consensus.cc:2229] T 2b1f589bca614c0c95b95b4304107784 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:15.626108 30671 raft_consensus.cc:2229] T 2b1f589bca614c0c95b95b4304107784 P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:15.626389 30672 raft_consensus.cc:2258] T 2b1f589bca614c0c95b95b4304107784 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:15.626667 30671 raft_consensus.cc:2258] T 2b1f589bca614c0c95b95b4304107784 P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:15.628957 30672 ts_tablet_manager.cc:1875] T 2b1f589bca614c0c95b95b4304107784 P e4b430a4ca81472ba0d5c42a4fa6f604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:02:15.628957 30671 ts_tablet_manager.cc:1875] T 2b1f589bca614c0c95b95b4304107784 P 747b41f97e7246cd8a42097546f44520: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:02:15.636180 30454 tablet_service.cc:1514] Processing DeleteTablet for tablet 661120bc29cf434e95c3f781d94d9106 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:02:15 UTC) from {username='slave'} at 127.0.0.1:56520
I20240501 20:02:15.636185 30455 tablet_service.cc:1514] Processing DeleteTablet for tablet e264a1b081524b36822b6b7d95969dba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:02:15 UTC) from {username='slave'} at 127.0.0.1:56520
W20240501 20:02:15.636808 30272 catalog_manager.cc:4573] TS 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b1f589bca614c0c95b95b4304107784: Network error: Client connection negotiation failed: client connection to 127.24.69.129:35925: connect: Connection refused (error 111)
I20240501 20:02:15.637655 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 20:02:15.646134 30672 ts_tablet_manager.cc:1888] T 2b1f589bca614c0c95b95b4304107784 P e4b430a4ca81472ba0d5c42a4fa6f604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240501 20:02:15.646729 30672 log.cc:1196] T 2b1f589bca614c0c95b95b4304107784 P e4b430a4ca81472ba0d5c42a4fa6f604: Deleting WAL directory at /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-1-root/wals/2b1f589bca614c0c95b95b4304107784
W20240501 20:02:15.647043 30674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:02:15.647567 30672 ts_tablet_manager.cc:1909] T 2b1f589bca614c0c95b95b4304107784 P e4b430a4ca81472ba0d5c42a4fa6f604: Deleting consensus metadata
W20240501 20:02:15.648325 30675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:02:15.649708 30671 ts_tablet_manager.cc:1888] T 2b1f589bca614c0c95b95b4304107784 P 747b41f97e7246cd8a42097546f44520: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240501 20:02:15.650091 30671 log.cc:1196] T 2b1f589bca614c0c95b95b4304107784 P 747b41f97e7246cd8a42097546f44520: Deleting WAL directory at /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-2-root/wals/2b1f589bca614c0c95b95b4304107784
I20240501 20:02:15.650751 30671 ts_tablet_manager.cc:1909] T 2b1f589bca614c0c95b95b4304107784 P 747b41f97e7246cd8a42097546f44520: Deleting consensus metadata
I20240501 20:02:15.651006 30272 catalog_manager.cc:4830] TS e4b430a4ca81472ba0d5c42a4fa6f604 (127.24.69.130:43685): tablet 2b1f589bca614c0c95b95b4304107784 (table dugtrio [id=c0373a073c56464ca05f378f58035f52]) successfully deleted
I20240501 20:02:15.651971 30672 tablet_replica.cc:330] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604: stopping tablet replica
I20240501 20:02:15.652936 30672 raft_consensus.cc:2229] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:15.653899 30671 tablet_replica.cc:330] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520: stopping tablet replica
I20240501 20:02:15.653970 30273 catalog_manager.cc:4830] TS 747b41f97e7246cd8a42097546f44520 (127.24.69.131:37749): tablet 2b1f589bca614c0c95b95b4304107784 (table dugtrio [id=c0373a073c56464ca05f378f58035f52]) successfully deleted
I20240501 20:02:15.653440 30672 raft_consensus.cc:2258] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:15.654845 30671 raft_consensus.cc:2229] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:02:15.655891 30671 raft_consensus.cc:2258] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:15.657263 30672 ts_tablet_manager.cc:1875] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240501 20:02:15.658428 30677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:02:15.658857 30671 ts_tablet_manager.cc:1875] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:02:15.659406 24854 server_base.cc:1034] running on GCE node
I20240501 20:02:15.660322 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:02:15.660530 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:02:15.660741 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593735660719 us; error 0 us; skew 500 ppm
I20240501 20:02:15.661319 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:02:15.664322 24854 webserver.cc:460] Webserver started at http://127.24.69.132:37987/ using document root <none> and password file <none>
I20240501 20:02:15.664774 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:02:15.664958 24854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:02:15.665206 24854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:02:15.666203 24854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-3-root/instance:
uuid: "4e06861ad21742af9cbbdd648abf12fa"
format_stamp: "Formatted at 2024-05-01 20:02:15 on dist-test-slave-z8x3"
I20240501 20:02:15.670926 24854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 20:02:15.671964 30672 ts_tablet_manager.cc:1888] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 20:02:15.672407 30672 log.cc:1196] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604: Deleting WAL directory at /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-1-root/wals/cf264333c5044dd9a9cd2036b1d8d54d
I20240501 20:02:15.673130 30672 ts_tablet_manager.cc:1909] T cf264333c5044dd9a9cd2036b1d8d54d P e4b430a4ca81472ba0d5c42a4fa6f604: Deleting consensus metadata
I20240501 20:02:15.673519 30671 ts_tablet_manager.cc:1888] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 20:02:15.673903 30671 log.cc:1196] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520: Deleting WAL directory at /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-2-root/wals/cf264333c5044dd9a9cd2036b1d8d54d
I20240501 20:02:15.674577 30682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:15.674626 30671 ts_tablet_manager.cc:1909] T cf264333c5044dd9a9cd2036b1d8d54d P 747b41f97e7246cd8a42097546f44520: Deleting consensus metadata
I20240501 20:02:15.675491 24854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:02:15.675832 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-3-root
uuid: "4e06861ad21742af9cbbdd648abf12fa"
format_stamp: "Formatted at 2024-05-01 20:02:15 on dist-test-slave-z8x3"
I20240501 20:02:15.676046 30672 tablet_replica.cc:330] T 661120bc29cf434e95c3f781d94d9106 P e4b430a4ca81472ba0d5c42a4fa6f604: stopping tablet replica
I20240501 20:02:15.676375 30272 catalog_manager.cc:4830] TS e4b430a4ca81472ba0d5c42a4fa6f604 (127.24.69.130:43685): tablet cf264333c5044dd9a9cd2036b1d8d54d (table dugtrio [id=c0373a073c56464ca05f378f58035f52]) successfully deleted
I20240501 20:02:15.676198 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:02:15.676975 30672 raft_consensus.cc:2229] T 661120bc29cf434e95c3f781d94d9106 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:15.677474 30672 raft_consensus.cc:2258] T 661120bc29cf434e95c3f781d94d9106 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:15.677872 30273 catalog_manager.cc:4830] TS 747b41f97e7246cd8a42097546f44520 (127.24.69.131:37749): tablet cf264333c5044dd9a9cd2036b1d8d54d (table dugtrio [id=c0373a073c56464ca05f378f58035f52]) successfully deleted
I20240501 20:02:15.678813 30671 tablet_replica.cc:330] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520: stopping tablet replica
I20240501 20:02:15.679224 30590 consensus_queue.cc:579] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [LEADER]: Leader has been unable to successfully communicate with peer 81885e2958c34da5ae5fe78b749d6e99 for more than 1 seconds (1.437s)
I20240501 20:02:15.679625 30671 raft_consensus.cc:2229] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:02:15.679819 30672 ts_tablet_manager.cc:1875] T 661120bc29cf434e95c3f781d94d9106 P e4b430a4ca81472ba0d5c42a4fa6f604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:02:15.680716 30671 raft_consensus.cc:2258] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:15.690380 30671 ts_tablet_manager.cc:1875] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:02:15.695258 30672 ts_tablet_manager.cc:1888] T 661120bc29cf434e95c3f781d94d9106 P e4b430a4ca81472ba0d5c42a4fa6f604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240501 20:02:15.695703 30672 log.cc:1196] T 661120bc29cf434e95c3f781d94d9106 P e4b430a4ca81472ba0d5c42a4fa6f604: Deleting WAL directory at /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-1-root/wals/661120bc29cf434e95c3f781d94d9106
I20240501 20:02:15.696367 30672 ts_tablet_manager.cc:1909] T 661120bc29cf434e95c3f781d94d9106 P e4b430a4ca81472ba0d5c42a4fa6f604: Deleting consensus metadata
I20240501 20:02:15.699117 30672 tablet_replica.cc:330] T e264a1b081524b36822b6b7d95969dba P e4b430a4ca81472ba0d5c42a4fa6f604: stopping tablet replica
I20240501 20:02:15.700002 30672 raft_consensus.cc:2229] T e264a1b081524b36822b6b7d95969dba P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:15.700567 30672 raft_consensus.cc:2258] T e264a1b081524b36822b6b7d95969dba P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:15.702359 30272 catalog_manager.cc:4830] TS e4b430a4ca81472ba0d5c42a4fa6f604 (127.24.69.130:43685): tablet 661120bc29cf434e95c3f781d94d9106 (table dugtrio [id=c0373a073c56464ca05f378f58035f52]) successfully deleted
I20240501 20:02:15.703799 30672 ts_tablet_manager.cc:1875] T e264a1b081524b36822b6b7d95969dba P e4b430a4ca81472ba0d5c42a4fa6f604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:02:15.710354 30671 ts_tablet_manager.cc:1888] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 20:02:15.710772 30671 log.cc:1196] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520: Deleting WAL directory at /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-2-root/wals/e264a1b081524b36822b6b7d95969dba
I20240501 20:02:15.711494 30671 ts_tablet_manager.cc:1909] T e264a1b081524b36822b6b7d95969dba P 747b41f97e7246cd8a42097546f44520: Deleting consensus metadata
I20240501 20:02:15.714322 30273 catalog_manager.cc:4830] TS 747b41f97e7246cd8a42097546f44520 (127.24.69.131:37749): tablet e264a1b081524b36822b6b7d95969dba (table dugtrio [id=c0373a073c56464ca05f378f58035f52]) successfully deleted
I20240501 20:02:15.714965 30671 tablet_replica.cc:330] T 661120bc29cf434e95c3f781d94d9106 P 747b41f97e7246cd8a42097546f44520: stopping tablet replica
I20240501 20:02:15.715801 30671 raft_consensus.cc:2229] T 661120bc29cf434e95c3f781d94d9106 P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:15.716259 30671 raft_consensus.cc:2258] T 661120bc29cf434e95c3f781d94d9106 P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:15.717406 30672 ts_tablet_manager.cc:1888] T e264a1b081524b36822b6b7d95969dba P e4b430a4ca81472ba0d5c42a4fa6f604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 20:02:15.717797 30672 log.cc:1196] T e264a1b081524b36822b6b7d95969dba P e4b430a4ca81472ba0d5c42a4fa6f604: Deleting WAL directory at /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-1-root/wals/e264a1b081524b36822b6b7d95969dba
I20240501 20:02:15.718472 30672 ts_tablet_manager.cc:1909] T e264a1b081524b36822b6b7d95969dba P e4b430a4ca81472ba0d5c42a4fa6f604: Deleting consensus metadata
I20240501 20:02:15.718479 30671 ts_tablet_manager.cc:1875] T 661120bc29cf434e95c3f781d94d9106 P 747b41f97e7246cd8a42097546f44520: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:02:15.720588 30272 catalog_manager.cc:4830] TS e4b430a4ca81472ba0d5c42a4fa6f604 (127.24.69.130:43685): tablet e264a1b081524b36822b6b7d95969dba (table dugtrio [id=c0373a073c56464ca05f378f58035f52]) successfully deleted
I20240501 20:02:15.731370 30671 ts_tablet_manager.cc:1888] T 661120bc29cf434e95c3f781d94d9106 P 747b41f97e7246cd8a42097546f44520: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240501 20:02:15.731773 30671 log.cc:1196] T 661120bc29cf434e95c3f781d94d9106 P 747b41f97e7246cd8a42097546f44520: Deleting WAL directory at /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-2-root/wals/661120bc29cf434e95c3f781d94d9106
I20240501 20:02:15.732401 30671 ts_tablet_manager.cc:1909] T 661120bc29cf434e95c3f781d94d9106 P 747b41f97e7246cd8a42097546f44520: Deleting consensus metadata
I20240501 20:02:15.735041 30273 catalog_manager.cc:4830] TS 747b41f97e7246cd8a42097546f44520 (127.24.69.131:37749): tablet 661120bc29cf434e95c3f781d94d9106 (table dugtrio [id=c0373a073c56464ca05f378f58035f52]) successfully deleted
I20240501 20:02:15.738909 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:02:15.739938 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:02:15.741485 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:02:15.744302 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:02:15.744493 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:15.744680 24854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:02:15.744807 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:15.752365 30661 raft_consensus.cc:482] T 6d8bde677f5e4f87802eda8373720358 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81885e2958c34da5ae5fe78b749d6e99)
I20240501 20:02:15.752964 30661 raft_consensus.cc:504] T 6d8bde677f5e4f87802eda8373720358 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:15.755261 30661 leader_election.cc:288] T 6d8bde677f5e4f87802eda8373720358 P e4b430a4ca81472ba0d5c42a4fa6f604 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925), 747b41f97e7246cd8a42097546f44520 (127.24.69.131:37749)
I20240501 20:02:15.756477 30539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8bde677f5e4f87802eda8373720358" candidate_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "747b41f97e7246cd8a42097546f44520" is_pre_election: true
W20240501 20:02:15.759059 30428 leader_election.cc:334] T 6d8bde677f5e4f87802eda8373720358 P e4b430a4ca81472ba0d5c42a4fa6f604 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925): Network error: Client connection negotiation failed: client connection to 127.24.69.129:35925: connect: Connection refused (error 111)
I20240501 20:02:15.759390 30428 leader_election.cc:302] T 6d8bde677f5e4f87802eda8373720358 P e4b430a4ca81472ba0d5c42a4fa6f604 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4b430a4ca81472ba0d5c42a4fa6f604; no voters: 747b41f97e7246cd8a42097546f44520, 81885e2958c34da5ae5fe78b749d6e99
I20240501 20:02:15.760033 30661 raft_consensus.cc:2726] T 6d8bde677f5e4f87802eda8373720358 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:02:15.761240 30590 raft_consensus.cc:482] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81885e2958c34da5ae5fe78b749d6e99)
I20240501 20:02:15.761693 30590 raft_consensus.cc:504] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:15.763212 30590 leader_election.cc:288] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925), e4b430a4ca81472ba0d5c42a4fa6f604 (127.24.69.130:43685)
I20240501 20:02:15.764629 30463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8bde677f5e4f87802eda8373720358" candidate_uuid: "747b41f97e7246cd8a42097546f44520" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" is_pre_election: true
I20240501 20:02:15.765278 30463 raft_consensus.cc:2454] T 6d8bde677f5e4f87802eda8373720358 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 747b41f97e7246cd8a42097546f44520 in term 1.
I20240501 20:02:15.766913 30502 leader_election.cc:302] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 747b41f97e7246cd8a42097546f44520, e4b430a4ca81472ba0d5c42a4fa6f604; no voters:
I20240501 20:02:15.767649 30590 raft_consensus.cc:2781] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240501 20:02:15.767696 30502 leader_election.cc:334] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925): Network error: Client connection negotiation failed: client connection to 127.24.69.129:35925: connect: Connection refused (error 111)
I20240501 20:02:15.768105 30590 raft_consensus.cc:482] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 81885e2958c34da5ae5fe78b749d6e99)
I20240501 20:02:15.768493 30590 raft_consensus.cc:3037] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:02:15.775544 30590 raft_consensus.cc:504] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
W20240501 20:02:15.781311 30502 leader_election.cc:334] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925): Network error: Client connection negotiation failed: client connection to 127.24.69.129:35925: connect: Connection refused (error 111)
I20240501 20:02:15.781497 30463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8bde677f5e4f87802eda8373720358" candidate_uuid: "747b41f97e7246cd8a42097546f44520" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604"
I20240501 20:02:15.781966 30590 leader_election.cc:288] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [CANDIDATE]: Term 2 election: Requested vote from peers 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925), e4b430a4ca81472ba0d5c42a4fa6f604 (127.24.69.130:43685)
I20240501 20:02:15.782112 30463 raft_consensus.cc:3037] T 6d8bde677f5e4f87802eda8373720358 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:02:15.788676 30463 raft_consensus.cc:2454] T 6d8bde677f5e4f87802eda8373720358 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 747b41f97e7246cd8a42097546f44520 in term 2.
I20240501 20:02:15.789645 30502 leader_election.cc:302] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 747b41f97e7246cd8a42097546f44520, e4b430a4ca81472ba0d5c42a4fa6f604; no voters: 81885e2958c34da5ae5fe78b749d6e99
I20240501 20:02:15.790450 30590 raft_consensus.cc:2781] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:02:15.791041 30590 raft_consensus.cc:686] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [term 2 LEADER]: Becoming Leader. State: Replica: 747b41f97e7246cd8a42097546f44520, State: Running, Role: LEADER
I20240501 20:02:15.791641 30590 consensus_queue.cc:237] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:15.803874 30288 catalog_manager.cc:5484] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 reported cstate change: term changed from 1 to 2, leader changed from 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129) to 747b41f97e7246cd8a42097546f44520 (127.24.69.131). New cstate: current_term: 2 leader_uuid: "747b41f97e7246cd8a42097546f44520" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } health_report { overall_health: HEALTHY } } }
I20240501 20:02:15.817091 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.132:33063
I20240501 20:02:15.817202 30745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.132:33063 every 8 connection(s)
I20240501 20:02:15.821285 24854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:02:15.828127 30749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:15.832360 30750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:02:15.832863 30746 heartbeater.cc:347] Connected to a master server at 127.24.69.190:40685
I20240501 20:02:15.833364 30746 heartbeater.cc:464] Registering TS with master...
I20240501 20:02:15.833432 24854 server_base.cc:1034] running on GCE node
W20240501 20:02:15.834051 30752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:02:15.834399 30746 heartbeater.cc:511] Master 127.24.69.190:40685 requested a full tablet report, sending...
I20240501 20:02:15.835088 24854 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:02:15.835343 24854 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:02:15.835507 24854 hybrid_clock.cc:648] HybridClock initialized: now 1714593735835491 us; error 0 us; skew 500 ppm
I20240501 20:02:15.836040 24854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:02:15.836915 30288 ts_manager.cc:194] Registered new tserver with Master: 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132:33063)
I20240501 20:02:15.838435 30288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41518
I20240501 20:02:15.841248 24854 webserver.cc:460] Webserver started at http://127.24.69.129:41731/ using document root <none> and password file <none>
I20240501 20:02:15.841728 24854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:02:15.841884 24854 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240501 20:02:15.845225 24854 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:02:15.848750 30757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:15.849495 24854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:02:15.849757 24854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-0-root
uuid: "81885e2958c34da5ae5fe78b749d6e99"
format_stamp: "Formatted at 2024-05-01 20:02:12 on dist-test-slave-z8x3"
I20240501 20:02:15.850032 24854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:02:15.864135 24854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:02:15.863875 30539 consensus_queue.cc:237] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } }
I20240501 20:02:15.865566 24854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:02:15.867405 24854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:02:15.871292 30465 raft_consensus.cc:1261] T 3378dd5a0ac64b3e852cb61ee61cec8b P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Refusing update from remote peer 747b41f97e7246cd8a42097546f44520: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:02:15.872865 30659 consensus_queue.cc:1035] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:02:15.886284 30767 ts_tablet_manager.cc:522] Loading tablet metadata (0/8 complete)
W20240501 20:02:15.886612 30502 consensus_peers.cc:485] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 -> Peer 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925): Couldn't send request to peer 81885e2958c34da5ae5fe78b749d6e99. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:35925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:02:15.889207 30503 consensus_peers.cc:485] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 -> Peer 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132:33063): Couldn't send request to peer 4e06861ad21742af9cbbdd648abf12fa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3378dd5a0ac64b3e852cb61ee61cec8b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:02:15.890856 30590 raft_consensus.cc:2932] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } } }
I20240501 20:02:15.893066 30465 raft_consensus.cc:2932] T 3378dd5a0ac64b3e852cb61ee61cec8b P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } } }
I20240501 20:02:15.902665 30539 consensus_queue.cc:237] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } }
I20240501 20:02:15.905246 30273 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3378dd5a0ac64b3e852cb61ee61cec8b with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 4)
I20240501 20:02:15.905472 30288 catalog_manager.cc:5484] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) added. New cstate: current_term: 1 leader_uuid: "747b41f97e7246cd8a42097546f44520" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:02:15.912590 30503 consensus_peers.cc:485] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 -> Peer 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132:33063): Couldn't send request to peer 4e06861ad21742af9cbbdd648abf12fa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dfd8f9eb29d745589b4aa63188a0c2aa. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:02:15.912876 30465 raft_consensus.cc:1261] T dfd8f9eb29d745589b4aa63188a0c2aa P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Refusing update from remote peer 747b41f97e7246cd8a42097546f44520: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:02:15.916317 30590 consensus_queue.cc:1035] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
W20240501 20:02:15.918323 30502 consensus_peers.cc:485] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 -> Peer 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925): Couldn't send request to peer 81885e2958c34da5ae5fe78b749d6e99. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:35925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:02:15.925365 30659 raft_consensus.cc:2932] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } } }
I20240501 20:02:15.926965 30465 raft_consensus.cc:2932] T dfd8f9eb29d745589b4aa63188a0c2aa P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } } }
I20240501 20:02:15.935648 30273 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dfd8f9eb29d745589b4aa63188a0c2aa with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 4)
I20240501 20:02:15.939535 30288 catalog_manager.cc:5484] T dfd8f9eb29d745589b4aa63188a0c2aa P e4b430a4ca81472ba0d5c42a4fa6f604 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) added. New cstate: current_term: 1 leader_uuid: "747b41f97e7246cd8a42097546f44520" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } } }
I20240501 20:02:15.953334 24854 ts_tablet_manager.cc:559] Loaded tablet metadata (8 total tablets, 8 live tablets)
I20240501 20:02:15.953796 24854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.076s user 0.009s sys 0.000s
I20240501 20:02:15.954205 24854 ts_tablet_manager.cc:574] Registering tablets (0/8 complete)
I20240501 20:02:15.961256 30767 tablet_bootstrap.cc:492] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap starting.
W20240501 20:02:15.967851 30336 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 81885e2958c34da5ae5fe78b749d6e99 not available for placement
I20240501 20:02:15.975001 30767 tablet_bootstrap.cc:492] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:02:15.975939 30767 tablet_bootstrap.cc:492] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap complete.
I20240501 20:02:15.976586 30767 ts_tablet_manager.cc:1367] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20240501 20:02:15.978868 30767 raft_consensus.cc:348] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:15.979733 30767 raft_consensus.cc:729] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81885e2958c34da5ae5fe78b749d6e99, State: Initialized, Role: FOLLOWER
I20240501 20:02:15.980391 30767 consensus_queue.cc:260] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:15.983095 30767 ts_tablet_manager.cc:1398] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 20:02:15.983923 30767 tablet_bootstrap.cc:492] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap starting.
I20240501 20:02:15.992328 30465 consensus_queue.cc:237] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } }
I20240501 20:02:15.998404 30767 tablet_bootstrap.cc:492] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:02:15.999338 30767 tablet_bootstrap.cc:492] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap complete.
I20240501 20:02:15.999985 30767 ts_tablet_manager.cc:1367] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.002s
I20240501 20:02:16.002202 24854 ts_tablet_manager.cc:590] Registered 8 tablets
I20240501 20:02:16.002506 24854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.048s user 0.045s sys 0.000s
I20240501 20:02:16.002369 30767 raft_consensus.cc:348] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:16.003162 30767 raft_consensus.cc:729] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81885e2958c34da5ae5fe78b749d6e99, State: Initialized, Role: FOLLOWER
I20240501 20:02:16.004472 30539 raft_consensus.cc:1261] T d3fb659b47994126bf3c60689f0b13b5 P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Refusing update from remote peer e4b430a4ca81472ba0d5c42a4fa6f604: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:02:16.004220 30767 consensus_queue.cc:260] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:16.007063 30767 ts_tablet_manager.cc:1398] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99: Time spent starting tablet: real 0.007s user 0.001s sys 0.004s
I20240501 20:02:16.008193 30767 tablet_bootstrap.cc:492] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap starting.
I20240501 20:02:16.008107 30661 consensus_queue.cc:1035] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
W20240501 20:02:16.016911 30428 consensus_peers.cc:485] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 -> Peer 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925): Couldn't send request to peer 81885e2958c34da5ae5fe78b749d6e99. Status: Network error: Client connection negotiation failed: client connection to 127.24.69.129:35925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:02:16.019678 30429 consensus_peers.cc:485] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 -> Peer 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132:33063): Couldn't send request to peer 4e06861ad21742af9cbbdd648abf12fa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3fb659b47994126bf3c60689f0b13b5. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:02:16.026608 30783 raft_consensus.cc:2932] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } } }
I20240501 20:02:16.028236 30539 raft_consensus.cc:2932] T d3fb659b47994126bf3c60689f0b13b5 P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } } }
I20240501 20:02:16.036864 30767 tablet_bootstrap.cc:492] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:02:16.037875 30767 tablet_bootstrap.cc:492] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap complete.
I20240501 20:02:16.038524 30767 ts_tablet_manager.cc:1367] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.013s
I20240501 20:02:16.039556 30272 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d3fb659b47994126bf3c60689f0b13b5 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
I20240501 20:02:16.041018 30767 raft_consensus.cc:348] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:16.041945 30767 raft_consensus.cc:729] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81885e2958c34da5ae5fe78b749d6e99, State: Initialized, Role: FOLLOWER
I20240501 20:02:16.042619 30767 consensus_queue.cc:260] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:16.042387 30288 catalog_manager.cc:5484] T d3fb659b47994126bf3c60689f0b13b5 P 747b41f97e7246cd8a42097546f44520 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) added. New cstate: current_term: 1 leader_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } } }
I20240501 20:02:16.044981 30767 ts_tablet_manager.cc:1398] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20240501 20:02:16.045817 30767 tablet_bootstrap.cc:492] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap starting.
I20240501 20:02:16.066200 30767 tablet_bootstrap.cc:492] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:02:16.067090 30767 tablet_bootstrap.cc:492] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap complete.
I20240501 20:02:16.067708 30767 ts_tablet_manager.cc:1367] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99: Time spent bootstrapping tablet: real 0.022s user 0.016s sys 0.004s
I20240501 20:02:16.069809 30767 raft_consensus.cc:348] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } }
I20240501 20:02:16.070518 30767 raft_consensus.cc:729] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81885e2958c34da5ae5fe78b749d6e99, State: Initialized, Role: FOLLOWER
I20240501 20:02:16.071125 30767 consensus_queue.cc:260] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } }
I20240501 20:02:16.072512 30767 ts_tablet_manager.cc:1398] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20240501 20:02:16.073166 30767 tablet_bootstrap.cc:492] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap starting.
I20240501 20:02:16.084148 30767 tablet_bootstrap.cc:492] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:02:16.085096 30767 tablet_bootstrap.cc:492] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap complete.
I20240501 20:02:16.085848 30767 ts_tablet_manager.cc:1367] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20240501 20:02:16.088454 30767 raft_consensus.cc:348] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:16.089252 30767 raft_consensus.cc:729] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81885e2958c34da5ae5fe78b749d6e99, State: Initialized, Role: FOLLOWER
I20240501 20:02:16.089931 30767 consensus_queue.cc:260] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:16.091913 30767 ts_tablet_manager.cc:1398] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:02:16.092746 30767 tablet_bootstrap.cc:492] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap starting.
I20240501 20:02:16.100947 24854 rpc_server.cc:307] RPC server started. Bound to: 127.24.69.129:35925
I20240501 20:02:16.100994 30838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.69.129:35925 every 8 connection(s)
I20240501 20:02:16.105036 30767 tablet_bootstrap.cc:492] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:02:16.105836 30767 tablet_bootstrap.cc:492] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap complete.
I20240501 20:02:16.106499 30767 ts_tablet_manager.cc:1367] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20240501 20:02:16.108398 30767 raft_consensus.cc:348] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:16.108862 30767 raft_consensus.cc:729] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81885e2958c34da5ae5fe78b749d6e99, State: Initialized, Role: FOLLOWER
I20240501 20:02:16.109382 30767 consensus_queue.cc:260] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:16.111048 30767 ts_tablet_manager.cc:1398] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 20:02:16.111698 30767 tablet_bootstrap.cc:492] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap starting.
I20240501 20:02:16.118213 30840 heartbeater.cc:347] Connected to a master server at 127.24.69.190:40685
I20240501 20:02:16.118655 30840 heartbeater.cc:464] Registering TS with master...
I20240501 20:02:16.119549 30840 heartbeater.cc:511] Master 127.24.69.190:40685 requested a full tablet report, sending...
I20240501 20:02:16.123847 30767 tablet_bootstrap.cc:492] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:02:16.124523 30767 tablet_bootstrap.cc:492] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap complete.
I20240501 20:02:16.124727 30288 ts_manager.cc:194] Re-registered known tserver with Master: 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925)
I20240501 20:02:16.124994 30767 ts_tablet_manager.cc:1367] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20240501 20:02:16.126566 30767 raft_consensus.cc:348] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:16.127009 30767 raft_consensus.cc:729] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81885e2958c34da5ae5fe78b749d6e99, State: Initialized, Role: FOLLOWER
I20240501 20:02:16.127511 30767 consensus_queue.cc:260] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:16.128834 30767 ts_tablet_manager.cc:1398] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 20:02:16.129516 30767 tablet_bootstrap.cc:492] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap starting.
I20240501 20:02:16.133189 30288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41524
I20240501 20:02:16.136059 30840 heartbeater.cc:503] Master 127.24.69.190:40685 was elected leader, sending a full tablet report...
I20240501 20:02:16.141144 30767 tablet_bootstrap.cc:492] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:02:16.141786 30767 tablet_bootstrap.cc:492] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99: Bootstrap complete.
I20240501 20:02:16.142239 30767 ts_tablet_manager.cc:1367] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20240501 20:02:16.143780 30767 raft_consensus.cc:348] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:16.144241 30767 raft_consensus.cc:729] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81885e2958c34da5ae5fe78b749d6e99, State: Initialized, Role: FOLLOWER
I20240501 20:02:16.144765 30767 consensus_queue.cc:260] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:16.145978 30767 ts_tablet_manager.cc:1398] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20240501 20:02:16.253197 30803 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1f589bca614c0c95b95b4304107784 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:02:15 UTC) from {username='slave'} at 127.0.0.1:57750
I20240501 20:02:16.254480 30846 tablet_replica.cc:330] stopping tablet replica
I20240501 20:02:16.255159 30846 raft_consensus.cc:2229] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:16.255542 30846 pending_rounds.cc:62] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99: Trying to abort 1 pending ops.
I20240501 20:02:16.255743 30846 pending_rounds.cc:68] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022975935664107520 op_type: NO_OP noop_request { }
I20240501 20:02:16.256031 30846 raft_consensus.cc:2866] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240501 20:02:16.256378 30846 raft_consensus.cc:2258] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:16.258394 30846 ts_tablet_manager.cc:1875] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:02:16.268369 30846 ts_tablet_manager.cc:1888] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 20:02:16.268659 30846 log.cc:1196] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99: Deleting WAL directory at /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-0-root/wals/2b1f589bca614c0c95b95b4304107784
I20240501 20:02:16.269260 30846 ts_tablet_manager.cc:1909] T 2b1f589bca614c0c95b95b4304107784 P 81885e2958c34da5ae5fe78b749d6e99: Deleting consensus metadata
I20240501 20:02:16.271559 30272 catalog_manager.cc:4830] TS 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925): tablet 2b1f589bca614c0c95b95b4304107784 (table dugtrio [id=c0373a073c56464ca05f378f58035f52]) successfully deleted
I20240501 20:02:16.280730 30814 raft_consensus.cc:2932] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } } }
I20240501 20:02:16.308596 30803 tablet_service.cc:1514] Processing DeleteTablet for tablet 661120bc29cf434e95c3f781d94d9106 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:02:15 UTC) from {username='slave'} at 127.0.0.1:57750
I20240501 20:02:16.309307 30846 tablet_replica.cc:330] stopping tablet replica
I20240501 20:02:16.309960 30846 raft_consensus.cc:2229] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:16.310341 30846 pending_rounds.cc:62] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99: Trying to abort 1 pending ops.
I20240501 20:02:16.310535 30846 pending_rounds.cc:68] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022975936273108992 op_type: NO_OP noop_request { }
I20240501 20:02:16.310837 30846 raft_consensus.cc:2866] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240501 20:02:16.311126 30846 raft_consensus.cc:2258] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:16.313264 30846 ts_tablet_manager.cc:1875] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:02:16.323271 30846 ts_tablet_manager.cc:1888] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 20:02:16.323555 30846 log.cc:1196] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99: Deleting WAL directory at /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-0-root/wals/661120bc29cf434e95c3f781d94d9106
I20240501 20:02:16.324301 30846 ts_tablet_manager.cc:1909] T 661120bc29cf434e95c3f781d94d9106 P 81885e2958c34da5ae5fe78b749d6e99: Deleting consensus metadata
I20240501 20:02:16.326603 30848 ts_tablet_manager.cc:902] T dfd8f9eb29d745589b4aa63188a0c2aa P 4e06861ad21742af9cbbdd648abf12fa: Initiating tablet copy from peer 747b41f97e7246cd8a42097546f44520 (127.24.69.131:37749)
I20240501 20:02:16.326633 30803 tablet_service.cc:1514] Processing DeleteTablet for tablet e264a1b081524b36822b6b7d95969dba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:02:15 UTC) from {username='slave'} at 127.0.0.1:57750
I20240501 20:02:16.326817 30272 catalog_manager.cc:4830] TS 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925): tablet 661120bc29cf434e95c3f781d94d9106 (table dugtrio [id=c0373a073c56464ca05f378f58035f52]) successfully deleted
I20240501 20:02:16.327600 30846 tablet_replica.cc:330] stopping tablet replica
I20240501 20:02:16.328460 30846 raft_consensus.cc:2229] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:16.328852 30846 raft_consensus.cc:2258] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:16.331081 30848 tablet_copy_client.cc:308] T dfd8f9eb29d745589b4aa63188a0c2aa P 4e06861ad21742af9cbbdd648abf12fa: tablet copy: Beginning tablet copy session from remote peer at address 127.24.69.131:37749
I20240501 20:02:16.331346 30846 ts_tablet_manager.cc:1875] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:02:16.345072 30803 tablet_service.cc:1514] Processing DeleteTablet for tablet cf264333c5044dd9a9cd2036b1d8d54d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:02:15 UTC) from {username='slave'} at 127.0.0.1:57750
I20240501 20:02:16.345108 30549 tablet_copy_service.cc:140] P 747b41f97e7246cd8a42097546f44520: Received BeginTabletCopySession request for tablet dfd8f9eb29d745589b4aa63188a0c2aa from peer 4e06861ad21742af9cbbdd648abf12fa ({username='slave'} at 127.0.0.1:37644)
I20240501 20:02:16.345824 30549 tablet_copy_service.cc:161] P 747b41f97e7246cd8a42097546f44520: Beginning new tablet copy session on tablet dfd8f9eb29d745589b4aa63188a0c2aa from peer 4e06861ad21742af9cbbdd648abf12fa at {username='slave'} at 127.0.0.1:37644: session id = 4e06861ad21742af9cbbdd648abf12fa-dfd8f9eb29d745589b4aa63188a0c2aa
I20240501 20:02:16.346451 30846 ts_tablet_manager.cc:1888] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240501 20:02:16.346814 30846 log.cc:1196] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99: Deleting WAL directory at /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-0-root/wals/e264a1b081524b36822b6b7d95969dba
I20240501 20:02:16.347411 30846 ts_tablet_manager.cc:1909] T e264a1b081524b36822b6b7d95969dba P 81885e2958c34da5ae5fe78b749d6e99: Deleting consensus metadata
I20240501 20:02:16.349776 30846 tablet_replica.cc:330] stopping tablet replica
I20240501 20:02:16.349769 30272 catalog_manager.cc:4830] TS 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925): tablet e264a1b081524b36822b6b7d95969dba (table dugtrio [id=c0373a073c56464ca05f378f58035f52]) successfully deleted
I20240501 20:02:16.350633 30846 raft_consensus.cc:2229] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:16.351089 30846 raft_consensus.cc:2258] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:16.351433 30549 tablet_copy_source_session.cc:205] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:02:16.353199 30846 ts_tablet_manager.cc:1875] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:02:16.354249 30848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfd8f9eb29d745589b4aa63188a0c2aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:16.367949 30846 ts_tablet_manager.cc:1888] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240501 20:02:16.368419 30846 log.cc:1196] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99: Deleting WAL directory at /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-0-root/wals/cf264333c5044dd9a9cd2036b1d8d54d
I20240501 20:02:16.368597 30848 tablet_copy_client.cc:783] T dfd8f9eb29d745589b4aa63188a0c2aa P 4e06861ad21742af9cbbdd648abf12fa: tablet copy: Starting download of 0 data blocks...
I20240501 20:02:16.369112 30846 ts_tablet_manager.cc:1909] T cf264333c5044dd9a9cd2036b1d8d54d P 81885e2958c34da5ae5fe78b749d6e99: Deleting consensus metadata
I20240501 20:02:16.369125 30848 tablet_copy_client.cc:647] T dfd8f9eb29d745589b4aa63188a0c2aa P 4e06861ad21742af9cbbdd648abf12fa: tablet copy: Starting download of 1 WAL segments...
I20240501 20:02:16.371671 30272 catalog_manager.cc:4830] TS 81885e2958c34da5ae5fe78b749d6e99 (127.24.69.129:35925): tablet cf264333c5044dd9a9cd2036b1d8d54d (table dugtrio [id=c0373a073c56464ca05f378f58035f52]) successfully deleted
I20240501 20:02:16.372596 30848 tablet_copy_client.cc:523] T dfd8f9eb29d745589b4aa63188a0c2aa P 4e06861ad21742af9cbbdd648abf12fa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:02:16.378654 30848 tablet_bootstrap.cc:492] T dfd8f9eb29d745589b4aa63188a0c2aa P 4e06861ad21742af9cbbdd648abf12fa: Bootstrap starting.
I20240501 20:02:16.394271 30848 tablet_bootstrap.cc:492] T dfd8f9eb29d745589b4aa63188a0c2aa P 4e06861ad21742af9cbbdd648abf12fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:02:16.395128 30848 tablet_bootstrap.cc:492] T dfd8f9eb29d745589b4aa63188a0c2aa P 4e06861ad21742af9cbbdd648abf12fa: Bootstrap complete.
I20240501 20:02:16.395829 30848 ts_tablet_manager.cc:1367] T dfd8f9eb29d745589b4aa63188a0c2aa P 4e06861ad21742af9cbbdd648abf12fa: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20240501 20:02:16.398698 30848 raft_consensus.cc:348] T dfd8f9eb29d745589b4aa63188a0c2aa P 4e06861ad21742af9cbbdd648abf12fa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } }
I20240501 20:02:16.399188 30465 raft_consensus.cc:1261] T 6d8bde677f5e4f87802eda8373720358 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 2 FOLLOWER]: Refusing update from remote peer 747b41f97e7246cd8a42097546f44520: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:02:16.399567 30848 raft_consensus.cc:729] T dfd8f9eb29d745589b4aa63188a0c2aa P 4e06861ad21742af9cbbdd648abf12fa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e06861ad21742af9cbbdd648abf12fa, State: Initialized, Role: LEARNER
I20240501 20:02:16.400233 30848 consensus_queue.cc:260] T dfd8f9eb29d745589b4aa63188a0c2aa P 4e06861ad21742af9cbbdd648abf12fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } }
I20240501 20:02:16.401302 30629 consensus_queue.cc:1035] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:02:16.403139 30812 raft_consensus.cc:3037] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:02:16.407750 30814 raft_consensus.cc:2932] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } } }
I20240501 20:02:16.428352 30861 ts_tablet_manager.cc:902] T 3378dd5a0ac64b3e852cb61ee61cec8b P 4e06861ad21742af9cbbdd648abf12fa: Initiating tablet copy from peer 747b41f97e7246cd8a42097546f44520 (127.24.69.131:37749)
I20240501 20:02:16.429126 30848 ts_tablet_manager.cc:1398] T dfd8f9eb29d745589b4aa63188a0c2aa P 4e06861ad21742af9cbbdd648abf12fa: Time spent starting tablet: real 0.033s user 0.014s sys 0.019s
I20240501 20:02:16.430243 30746 heartbeater.cc:503] Master 127.24.69.190:40685 was elected leader, sending a full tablet report...
I20240501 20:02:16.431319 30549 tablet_copy_service.cc:342] P 747b41f97e7246cd8a42097546f44520: Request end of tablet copy session 4e06861ad21742af9cbbdd648abf12fa-dfd8f9eb29d745589b4aa63188a0c2aa received from {username='slave'} at 127.0.0.1:37644
I20240501 20:02:16.431741 30549 tablet_copy_service.cc:433] P 747b41f97e7246cd8a42097546f44520: ending tablet copy session 4e06861ad21742af9cbbdd648abf12fa-dfd8f9eb29d745589b4aa63188a0c2aa on tablet dfd8f9eb29d745589b4aa63188a0c2aa with peer 4e06861ad21742af9cbbdd648abf12fa
I20240501 20:02:16.449220 30848 ts_tablet_manager.cc:902] T d3fb659b47994126bf3c60689f0b13b5 P 4e06861ad21742af9cbbdd648abf12fa: Initiating tablet copy from peer e4b430a4ca81472ba0d5c42a4fa6f604 (127.24.69.130:43685)
F20240501 20:02:16.451082 30848 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-128e1005f1df4010982d9ebf0c02f727.queue_time_us, tablet-download-pool-128e1005f1df4010982d9ebf0c02f727.queue_time_us) (tablet-download-pool-128e1005f1df4010982d9ebf0c02f727.run_cpu_time_us, tablet-download-pool-128e1005f1df4010982d9ebf0c02f727.run_cpu_time_us) (tablet-download-pool-128e1005f1df4010982d9ebf0c02f727.run_wall_time_us, tablet-download-pool-128e1005f1df4010982d9ebf0c02f727.run_wall_time_us) (tablet-download-pool-1a6bd1d77970465487acd1abe91687ad.queue_time_us, tablet-download-pool-1a6bd1d77970465487acd1abe91687ad.queue_time_us) (tablet-download-pool-1a6bd1d77970465487acd1abe91687ad.run_cpu_time_us, tablet-download-pool-1a6bd1d77970465487acd1abe91687ad.run_cpu_time_us) (tablet-download-pool-1a6bd1d77970465487acd1abe91687ad.run_wall_time_us, tablet-download-pool-1a6bd1d77970465487acd1abe91687ad.run_wall_time_us) (tablet-download-pool-2587cb164a134d5f9112e11a7176e223.queue_time_us, tablet-download-pool-2587cb164a134d5f9112e11a7176e223.queue_time_us) (tablet-download-pool-2587cb164a134d5f9112e11a7176e223.run_cpu_time_us, tablet-download-pool-2587cb164a134d5f9112e11a7176e223.run_cpu_time_us) (tablet-download-pool-2587cb164a134d5f9112e11a7176e223.run_wall_time_us, tablet-download-pool-2587cb164a134d5f9112e11a7176e223.run_wall_time_us) (tablet-download-pool-268d32d69f8d4591b72359673a8d9b92.queue_time_us, tablet-download-pool-268d32d69f8d4591b72359673a8d9b92.queue_time_us) (tablet-download-pool-268d32d69f8d4591b72359673a8d9b92.run_cpu_time_us, tablet-download-pool-268d32d69f8d4591b72359673a8d9b92.run_cpu_time_us) (tablet-download-pool-268d32d69f8d4591b72359673a8d9b92.run_wall_time_us, tablet-download-pool-268d32d69f8d4591b72359673a8d9b92.run_wall_time_us) (tablet-download-pool-27b9198949354d2e9202677a25193fa9.queue_time_us, tablet-download-pool-27b9198949354d2e9202677a25193fa9.queue_time_us) (tablet-download-pool-27b9198949354d2e9202677a25193fa9.run_cpu_time_us, tablet-download-pool-27b9198949354d2e9202677a25193fa9.run_cpu_time_us) (tablet-download-pool-27b9198949354d2e9202677a25193fa9.run_wall_time_us, tablet-download-pool-27b9198949354d2e9202677a25193fa9.run_wall_time_us) (tablet-download-pool-3378dd5a0ac64b3e852cb61ee61cec8b.queue_time_us, tablet-download-pool-3378dd5a0ac64b3e852cb61ee61cec8b.queue_time_us) (tablet-download-pool-3378dd5a0ac64b3e852cb61ee61cec8b.run_cpu_time_us, tablet-download-pool-3378dd5a0ac64b3e852cb61ee61cec8b.run_cpu_time_us) (tablet-download-pool-3378dd5a0ac64b3e852cb61ee61cec8b.run_wall_time_us, tablet-download-pool-3378dd5a0ac64b3e852cb61ee61cec8b.run_wall_time_us) (tablet-download-pool-82aaf53585ea4d87aa1d65dc40658d3f.queue_time_us, tablet-download-pool-82aaf53585ea4d87aa1d65dc40658d3f.queue_time_us) (tablet-download-pool-82aaf53585ea4d87aa1d65dc40658d3f.run_cpu_time_us, tablet-download-pool-82aaf53585ea4d87aa1d65dc40658d3f.run_cpu_time_us) (tablet-download-pool-82aaf53585ea4d87aa1d65dc40658d3f.run_wall_time_us, tablet-download-pool-82aaf53585ea4d87aa1d65dc40658d3f.run_wall_time_us) (tablet-download-pool-96a2762d729f41a3a34b013bf76d4702.queue_time_us, tablet-download-pool-96a2762d729f41a3a34b013bf76d4702.queue_time_us) (tablet-download-pool-96a2762d729f41a3a34b013bf76d4702.run_cpu_time_us, tablet-download-pool-96a2762d729f41a3a34b013bf76d4702.run_cpu_time_us) (tablet-download-pool-96a2762d729f41a3a34b013bf76d4702.run_wall_time_us, tablet-download-pool-96a2762d729f41a3a34b013bf76d4702.run_wall_time_us) (tablet-download-pool-bd534f7d16d44a478ec5f954776b93aa.queue_time_us, tablet-download-pool-bd534f7d16d44a478ec5f954776b93aa.queue_time_us) (tablet-download-pool-bd534f7d16d44a478ec5f954776b93aa.run_cpu_time_us, tablet-download-pool-bd534f7d16d44a478ec5f954776b93aa.run_cpu_time_us) (tablet-download-pool-bd534f7d16d44a478ec5f954776b93aa.run_wall_time_us, tablet-download-pool-bd534f7d16d44a478ec5f954776b93aa.run_wall_time_us) (tablet-download-pool-cd88640130a342d899a02a89c3a9fcfa.queue_time_us, tablet-download-pool-cd88640130a342d899a02a89c3a9fcfa.queue_time_us) (tablet-download-pool-cd88640130a342d899a02a89c3a9fcfa.run_cpu_time_us, tablet-download-pool-cd88640130a342d899a02a89c3a9fcfa.run_cpu_time_us) (tablet-download-pool-cd88640130a342d899a02a89c3a9fcfa.run_wall_time_us, tablet-download-pool-cd88640130a342d899a02a89c3a9fcfa.run_wall_time_us) (tablet-download-pool-d3fb659b47994126bf3c60689f0b13b5.queue_time_us, tablet-download-pool-d3fb659b47994126bf3c60689f0b13b5.queue_time_us) (tablet-download-pool-dfd8f9eb29d745589b4aa63188a0c2aa.queue_time_us, tablet-download-pool-dfd8f9eb29d745589b4aa63188a0c2aa.queue_time_us) (tablet-download-pool-dfd8f9eb29d745589b4aa63188a0c2aa.run_cpu_time_us, tablet-download-pool-dfd8f9eb29d745589b4aa63188a0c2aa.run_cpu_time_us) (tablet-download-pool-dfd8f9eb29d745589b4aa63188a0c2aa.run_wall_time_us, tablet-download-pool-dfd8f9eb29d745589b4aa63188a0c2aa.run_wall_time_us) (tablet-download-pool-ed6ad07388d449cf8a8504a00f60dbd8.queue_time_us, tablet-download-pool-ed6ad07388d449cf8a8504a00f60dbd8.queue_time_us) (tablet-download-pool-ed6ad07388d449cf8a8504a00f60dbd8.run_cpu_time_us, tablet-download-pool-ed6ad07388d449cf8a8504a00f60dbd8.run_cpu_time_us) (tablet-download-pool-ed6ad07388d449cf8a8504a00f60dbd8.run_wall_time_us, tablet-download-pool-ed6ad07388d449cf8a8504a00f60dbd8.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240501 20:02:16.460601 30812 raft_consensus.cc:1261] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [term 2 FOLLOWER]: Refusing update from remote peer 747b41f97e7246cd8a42097546f44520: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:02:16.463259 30814 raft_consensus.cc:2932] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } } }
I20240501 20:02:16.473570 30629 consensus_queue.cc:1035] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
*** Check failure stack trace: ***
*** Aborted at 1714593736 (unix time) try "date -d @1714593736" if you are using GNU date ***
I20240501 20:02:16.501235 30861 tablet_copy_client.cc:308] T 3378dd5a0ac64b3e852cb61ee61cec8b P 4e06861ad21742af9cbbdd648abf12fa: tablet copy: Beginning tablet copy session from remote peer at address 127.24.69.131:37749
I20240501 20:02:16.502979 30549 tablet_copy_service.cc:140] P 747b41f97e7246cd8a42097546f44520: Received BeginTabletCopySession request for tablet 3378dd5a0ac64b3e852cb61ee61cec8b from peer 4e06861ad21742af9cbbdd648abf12fa ({username='slave'} at 127.0.0.1:37644)
I20240501 20:02:16.503433 30549 tablet_copy_service.cc:161] P 747b41f97e7246cd8a42097546f44520: Beginning new tablet copy session on tablet 3378dd5a0ac64b3e852cb61ee61cec8b from peer 4e06861ad21742af9cbbdd648abf12fa at {username='slave'} at 127.0.0.1:37644: session id = 4e06861ad21742af9cbbdd648abf12fa-3378dd5a0ac64b3e852cb61ee61cec8b
I20240501 20:02:16.509692 30549 tablet_copy_source_session.cc:205] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:02:16.513135 30861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3378dd5a0ac64b3e852cb61ee61cec8b. 1 dirs total, 0 dirs full, 0 dirs failed
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800006116) received by PID 24854 (TID 0x7fabf1ae6700) from PID 24854; stack trace: ***
@ 0x2e41a0 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2e6683 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
I20240501 20:02:16.529742 30861 tablet_copy_client.cc:783] T 3378dd5a0ac64b3e852cb61ee61cec8b P 4e06861ad21742af9cbbdd648abf12fa: tablet copy: Starting download of 0 data blocks...
I20240501 20:02:16.530504 30861 tablet_copy_client.cc:647] T 3378dd5a0ac64b3e852cb61ee61cec8b P 4e06861ad21742af9cbbdd648abf12fa: tablet copy: Starting download of 1 WAL segments...
I20240501 20:02:16.534098 30861 tablet_copy_client.cc:523] T 3378dd5a0ac64b3e852cb61ee61cec8b P 4e06861ad21742af9cbbdd648abf12fa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7faca2061980 (unknown) at ??:0
I20240501 20:02:16.543819 30861 tablet_bootstrap.cc:492] T 3378dd5a0ac64b3e852cb61ee61cec8b P 4e06861ad21742af9cbbdd648abf12fa: Bootstrap starting.
@ 0x7fac9b5d5fb7 gsignal at ??:0
@ 0x7fac9b5d7921 abort at ??:0
@ 0x2ea196 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7fac9e3cbda4 google::LogMessage::SendToLog() at ??:0
@ 0x7fac9e3cc950 google::LogMessage::Flush() at ??:0
@ 0x7fac9e3d1a8b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fac9ed593c7 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fac9ed2f94f kudu::ThreadPool::ThreadPool() at ??:0
I20240501 20:02:16.689307 30539 consensus_queue.cc:237] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } }
I20240501 20:02:16.694221 30465 raft_consensus.cc:1261] T 3378dd5a0ac64b3e852cb61ee61cec8b P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Refusing update from remote peer 747b41f97e7246cd8a42097546f44520: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:02:16.694471 30814 raft_consensus.cc:1261] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Refusing update from remote peer 747b41f97e7246cd8a42097546f44520: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
@ 0x7fac9ed2d0a5 kudu::ThreadPoolBuilder::Build() at ??:0
I20240501 20:02:16.695809 30659 consensus_queue.cc:1035] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:02:16.696516 30629 consensus_queue.cc:1035] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:02:16.702477 30771 raft_consensus.cc:2932] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } }
I20240501 20:02:16.708967 30273 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3378dd5a0ac64b3e852cb61ee61cec8b with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:02:16.712391 30814 raft_consensus.cc:2932] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } }
I20240501 20:02:16.712920 30465 raft_consensus.cc:2932] T 3378dd5a0ac64b3e852cb61ee61cec8b P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } }
I20240501 20:02:16.713794 30285 catalog_manager.cc:5484] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 reported cstate change: config changed from index 2 to 3, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) evicted. New cstate: current_term: 1 leader_uuid: "747b41f97e7246cd8a42097546f44520" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } health_report { overall_health: UNKNOWN } } }
I20240501 20:02:16.725363 30720 raft_consensus.cc:1203] T dfd8f9eb29d745589b4aa63188a0c2aa P 4e06861ad21742af9cbbdd648abf12fa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
@ 0x7faca6b4d422 kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240501 20:02:16.750413 30710 tablet_service.cc:1514] Processing DeleteTablet for tablet 3378dd5a0ac64b3e852cb61ee61cec8b with delete_type TABLET_DATA_TOMBSTONED (TS 4e06861ad21742af9cbbdd648abf12fa not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53112
W20240501 20:02:16.753131 30273 catalog_manager.cc:4810] TS 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132:33063): delete failed for tablet 3378dd5a0ac64b3e852cb61ee61cec8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3378dd5a0ac64b3e852cb61ee61cec8b already in progress: copying tablet
@ 0x7faca6b53c89 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7faca6be94ef kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7faca6c1a3cf kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7faca6c001ce kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7faca6c0017a _ZNSt3__18__invokeIRZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBENS_8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEE4$_12JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSH_DpOSI_ at ??:0
@ 0x7faca6c0010a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7faca6c000d2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7faca6bff2fe std::__1::__function::__func<>::operator()() at ??:0
@ 0x7faca7fa5da5 std::__1::__function::__value_func<>::operator()() at ??:0
I20240501 20:02:16.969475 30465 consensus_queue.cc:237] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
@ 0x7faca7fa5bda std::__1::function<>::operator()() at ??:0
I20240501 20:02:16.978183 30539 raft_consensus.cc:1261] T d3fb659b47994126bf3c60689f0b13b5 P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Refusing update from remote peer e4b430a4ca81472ba0d5c42a4fa6f604: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:02:16.979156 30814 raft_consensus.cc:1261] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Refusing update from remote peer e4b430a4ca81472ba0d5c42a4fa6f604: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:02:16.980235 30661 consensus_queue.cc:1035] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:02:16.981937 30661 consensus_queue.cc:1035] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:02:17.002027 30901 raft_consensus.cc:2932] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } }
I20240501 20:02:17.003979 30814 raft_consensus.cc:2932] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } }
@ 0x7fac9ed31237 kudu::ThreadPool::DispatchThread() at ??:0
I20240501 20:02:17.021184 30272 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d3fb659b47994126bf3c60689f0b13b5 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:02:17.024050 30539 consensus_queue.cc:237] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } }
I20240501 20:02:17.027351 30538 raft_consensus.cc:2932] T d3fb659b47994126bf3c60689f0b13b5 P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } }
I20240501 20:02:17.035569 30465 raft_consensus.cc:1261] T dfd8f9eb29d745589b4aa63188a0c2aa P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Refusing update from remote peer 747b41f97e7246cd8a42097546f44520: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:02:17.035753 30814 raft_consensus.cc:1261] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Refusing update from remote peer 747b41f97e7246cd8a42097546f44520: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:02:17.037513 30881 consensus_queue.cc:1035] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:02:17.039024 30881 consensus_queue.cc:1035] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
@ 0x7fac9ed34692 kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20240501 20:02:17.040515 30288 catalog_manager.cc:5484] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99 reported cstate change: config changed from index 2 to 3, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) evicted. New cstate: current_term: 1 leader_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } }
I20240501 20:02:17.049597 30771 raft_consensus.cc:2932] T dfd8f9eb29d745589b4aa63188a0c2aa P 747b41f97e7246cd8a42097546f44520 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } }
I20240501 20:02:17.062336 30814 raft_consensus.cc:2932] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } }
I20240501 20:02:17.067806 30463 consensus_queue.cc:237] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } attrs { replace: true } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } }
@ 0x7fac9ed3464a _ZNSt3__18__invokeIRZN4kudu10ThreadPool12CreateThreadEvE3$_2JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS5_DpOS6_ at ??:0
I20240501 20:02:17.078241 30710 tablet_service.cc:1514] Processing DeleteTablet for tablet d3fb659b47994126bf3c60689f0b13b5 with delete_type TABLET_DATA_TOMBSTONED (TS 4e06861ad21742af9cbbdd648abf12fa not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53112
W20240501 20:02:17.078632 30429 consensus_peers.cc:485] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 -> Peer 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132:33063): Couldn't send request to peer 4e06861ad21742af9cbbdd648abf12fa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3fb659b47994126bf3c60689f0b13b5. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:02:17.082599 30465 raft_consensus.cc:2932] T dfd8f9eb29d745589b4aa63188a0c2aa P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } }
W20240501 20:02:17.083690 30273 catalog_manager.cc:4797] TS 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132:33063): delete failed for tablet d3fb659b47994126bf3c60689f0b13b5 because the tablet was not found. No further retry: Not found: Tablet not found: d3fb659b47994126bf3c60689f0b13b5
I20240501 20:02:17.085585 30538 raft_consensus.cc:1261] T d3fb659b47994126bf3c60689f0b13b5 P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Refusing update from remote peer e4b430a4ca81472ba0d5c42a4fa6f604: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:02:17.087145 30901 consensus_queue.cc:1035] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:02:17.088058 30814 raft_consensus.cc:1261] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Refusing update from remote peer e4b430a4ca81472ba0d5c42a4fa6f604: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:02:17.089969 30901 consensus_queue.cc:1035] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:02:17.095084 30273 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dfd8f9eb29d745589b4aa63188a0c2aa with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:02:17.095891 30287 catalog_manager.cc:5484] T dfd8f9eb29d745589b4aa63188a0c2aa P 81885e2958c34da5ae5fe78b749d6e99 reported cstate change: config changed from index 2 to 3, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) evicted. New cstate: current_term: 1 leader_uuid: "747b41f97e7246cd8a42097546f44520" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } }
I20240501 20:02:17.099411 30661 raft_consensus.cc:2932] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } attrs { replace: true } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } } }
@ 0x7fac9ed345da std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20240501 20:02:17.106170 30538 raft_consensus.cc:2932] T d3fb659b47994126bf3c60689f0b13b5 P 747b41f97e7246cd8a42097546f44520 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } attrs { replace: true } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } } }
I20240501 20:02:17.117451 30285 catalog_manager.cc:5484] T d3fb659b47994126bf3c60689f0b13b5 P e4b430a4ca81472ba0d5c42a4fa6f604 reported cstate change: config changed from index 3 to 4, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) added. New cstate: current_term: 1 leader_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:02:17.102411 30814 raft_consensus.cc:2932] T d3fb659b47994126bf3c60689f0b13b5 P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } attrs { replace: true } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } } }
I20240501 20:02:17.129478 30710 tablet_service.cc:1514] Processing DeleteTablet for tablet dfd8f9eb29d745589b4aa63188a0c2aa with delete_type TABLET_DATA_TOMBSTONED (TS 4e06861ad21742af9cbbdd648abf12fa not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:53112
I20240501 20:02:17.130776 30887 tablet_replica.cc:330] T dfd8f9eb29d745589b4aa63188a0c2aa P 4e06861ad21742af9cbbdd648abf12fa: stopping tablet replica
I20240501 20:02:17.131647 30887 raft_consensus.cc:2229] T dfd8f9eb29d745589b4aa63188a0c2aa P 4e06861ad21742af9cbbdd648abf12fa [term 1 LEARNER]: Raft consensus shutting down.
I20240501 20:02:17.132392 30887 raft_consensus.cc:2258] T dfd8f9eb29d745589b4aa63188a0c2aa P 4e06861ad21742af9cbbdd648abf12fa [term 1 LEARNER]: Raft consensus is shut down!
I20240501 20:02:17.139140 30887 ts_tablet_manager.cc:1875] T dfd8f9eb29d745589b4aa63188a0c2aa P 4e06861ad21742af9cbbdd648abf12fa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
@ 0x7fac9ed345a2 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20240501 20:02:17.160296 30538 consensus_queue.cc:237] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } attrs { replace: true } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } }
I20240501 20:02:17.165549 30887 ts_tablet_manager.cc:1888] T dfd8f9eb29d745589b4aa63188a0c2aa P 4e06861ad21742af9cbbdd648abf12fa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 20:02:17.165930 30465 raft_consensus.cc:1261] T 3378dd5a0ac64b3e852cb61ee61cec8b P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Refusing update from remote peer 747b41f97e7246cd8a42097546f44520: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:02:17.166118 30887 log.cc:1196] T dfd8f9eb29d745589b4aa63188a0c2aa P 4e06861ad21742af9cbbdd648abf12fa: Deleting WAL directory at /tmp/dist-test-taskNlxqne/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1714593576104309-24854-0/minicluster-data/ts-3-root/wals/dfd8f9eb29d745589b4aa63188a0c2aa
I20240501 20:02:17.167985 30771 consensus_queue.cc:1035] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:02:17.169595 30273 catalog_manager.cc:4830] TS 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132:33063): tablet dfd8f9eb29d745589b4aa63188a0c2aa (table test-workload [id=331b4d80f40e4828a7954cbafaf2e5e2]) successfully deleted
W20240501 20:02:17.171502 30503 consensus_peers.cc:485] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 -> Peer 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132:33063): Couldn't send request to peer 4e06861ad21742af9cbbdd648abf12fa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:02:17.172719 30814 raft_consensus.cc:1261] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Refusing update from remote peer 747b41f97e7246cd8a42097546f44520: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
@ 0x7fac9ed3389e std::__1::__function::__func<>::operator()() at ??:0
I20240501 20:02:17.174404 30881 consensus_queue.cc:1035] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:02:17.176813 30771 raft_consensus.cc:2932] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } attrs { replace: true } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } } }
I20240501 20:02:17.181887 30465 raft_consensus.cc:2932] T 3378dd5a0ac64b3e852cb61ee61cec8b P e4b430a4ca81472ba0d5c42a4fa6f604 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } attrs { replace: true } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } } }
I20240501 20:02:17.189505 30814 raft_consensus.cc:2932] T 3378dd5a0ac64b3e852cb61ee61cec8b P 81885e2958c34da5ae5fe78b749d6e99 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } attrs { replace: true } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } } }
I20240501 20:02:17.190178 30288 catalog_manager.cc:5484] T 3378dd5a0ac64b3e852cb61ee61cec8b P 747b41f97e7246cd8a42097546f44520 reported cstate change: config changed from index 3 to 4, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) added. New cstate: current_term: 1 leader_uuid: "747b41f97e7246cd8a42097546f44520" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:02:17.195627 30538 consensus_queue.cc:237] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } attrs { replace: true } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } }
@ 0x7faca7fa5da5 std::__1::__function::__value_func<>::operator()() at ??:0
I20240501 20:02:17.208853 30465 raft_consensus.cc:1261] T 6d8bde677f5e4f87802eda8373720358 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 2 FOLLOWER]: Refusing update from remote peer 747b41f97e7246cd8a42097546f44520: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20240501 20:02:17.210184 30503 consensus_peers.cc:485] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 -> Peer 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132:33063): Couldn't send request to peer 4e06861ad21742af9cbbdd648abf12fa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d8bde677f5e4f87802eda8373720358. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:02:17.210839 30881 consensus_queue.cc:1035] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:02:17.211555 30812 raft_consensus.cc:1261] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [term 2 FOLLOWER]: Refusing update from remote peer 747b41f97e7246cd8a42097546f44520: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 20:02:17.213500 30881 consensus_queue.cc:1035] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:02:17.220319 30629 raft_consensus.cc:2932] T 6d8bde677f5e4f87802eda8373720358 P 747b41f97e7246cd8a42097546f44520 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } attrs { replace: true } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } } }
I20240501 20:02:17.221980 30465 raft_consensus.cc:2932] T 6d8bde677f5e4f87802eda8373720358 P e4b430a4ca81472ba0d5c42a4fa6f604 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } attrs { replace: true } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } } }
I20240501 20:02:17.223292 30814 raft_consensus.cc:2932] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } attrs { replace: true } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } } }
@ 0x7faca7fa5bda std::__1::function<>::operator()() at ??:0
I20240501 20:02:17.237169 30285 catalog_manager.cc:5484] T 6d8bde677f5e4f87802eda8373720358 P 81885e2958c34da5ae5fe78b749d6e99 reported cstate change: config changed from index -1 to 3, NON_VOTER 4e06861ad21742af9cbbdd648abf12fa (127.24.69.132) added. New cstate: current_term: 2 leader_uuid: "747b41f97e7246cd8a42097546f44520" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "81885e2958c34da5ae5fe78b749d6e99" member_type: VOTER last_known_addr { host: "127.24.69.129" port: 35925 } attrs { replace: true } } peers { permanent_uuid: "e4b430a4ca81472ba0d5c42a4fa6f604" member_type: VOTER last_known_addr { host: "127.24.69.130" port: 43685 } } peers { permanent_uuid: "747b41f97e7246cd8a42097546f44520" member_type: VOTER last_known_addr { host: "127.24.69.131" port: 37749 } } peers { permanent_uuid: "4e06861ad21742af9cbbdd648abf12fa" member_type: NON_VOTER last_known_addr { host: "127.24.69.132" port: 33063 } attrs { promote: true } } }
@ 0x7fac9ed13937 kudu::Thread::SuperviseThread() at ??:0